Incus VM in Hyper-V nested virtualization crashes when memory limit greater than 2815MiB is set

Incus
1

Hi everyone,

I’ve encountered a bug when running nested VMs via Incus (6.0.4) on Debian Trixie in Hyper-V (Version 10.0.26100.5074) on Windows 11.

I get the following Error:

Code=qemu-system-x86_64: hw/core/cpu-system.c:90: cpu_asidx_from_attrs: Assertion `ret < cpu->num_ases && ret >= 0' failed.

Reproducer

Create a new Gen2 Hyper-V VM with the following settings

  • 20GB of memory (just to be sure it’s not due to OOM)

  • disabled dynamic memory

  • disabled secure boot

  • enable nested virtualization (As admin: Set-VMProcessor -VMName "Trixie Crash Test" -ExposeVirtualizationExtensions $true)

Then, install Debian 13 Trixie with all default values, for features only SSH Server + standard system utilities only.

In that new VM, I then ran as root:

apt update && apt upgrade
apt install incus
# Choose the default for everything
incus admin init
incus init images:debian/13 debian-test --vm -c limits.memory=8GB
incus start debian-test --console
# You will be thrown back to your root prompt after a few seconds
# Then, you find the error from above using
incus info debian-test --show-log

Error:

Name: debian-test
Description: 
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: 2025/10/06 10:34 PDT
Last Used: 2025/10/06 10:34 PDT

Log:

KVM: entry failed, hardware error 0xffffffff
EAX=00000000 EBX=94003d38 ECX=000000b2 EDX=000000b2
ESI=00000000 EDI=00000058 EBP=94003ca0 ESP=94003c78
EIP=00008000 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=1 HLT=0
ES =0000 00000000 ffffffff 00809300
CS =f900 7eff9000 ffffffff 00809300
SS =0000 00000000 ffffffff 00809300
DS =0000 00000000 ffffffff 00809300
FS =0000 00000000 ffffffff 00809300
GS =0000 00000000 ffffffff 00809300
LDT=0000 00000000 00000000 00000000
TR =0040 04a2b000 00004087 00008b00
GDT=     04a29000 0000007f
IDT=     00000000 00000000
CR0=00050032 CR2=b2a01000 CR3=00228000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=qemu-system-x86_64: hw/core/cpu-system.c:90: cpu_asidx_from_attrs: Assertion `ret < cpu->num_ases && ret >= 0' failed.

I’ve then tried to narrow down the maximum amount of Memory I can give the machine using binary search and arrived at 2815MiB

incus config set debian-test limits.memory 2815MiB
incus start debian-test --console
# Works, you get the debian serial console login prompt

incus config set debian-test limits.memory 2816MiB
incus start debian-test --console
# Does not work, thrown back to root shell

The hardware I am running this on is the following:

         Operating System: Windows 11 Pro 64-bit (10.0, Build 26100) (26100.ge_release.240331-1435)
                 Language: English (Regional Setting: English)
      System Manufacturer: Gigabyte Technology Co., Ltd.
             System Model: X570 AORUS ELITE
                     BIOS: F37 (type: UEFI)
                Processor: AMD Ryzen 7 3800X 8-Core Processor              (16 CPUs), ~3.9GHz
                   Memory: 65536MB RAM
      Available OS Memory: 65464MB RAM
                Page File: 63623MB used, 5936MB available

The Question

Is this a known issue, and if not, would you be open to helping me debug this? I can of course provide more informationand offer my help as well :slight_smile:

2

Not sure what’s causing this, although I know over the summer there was some work in improving/fixing the memory hotplugging logic that wouldn’t have been in 6.0.4. From the error that’s the first thing that came to mind. I know larger memory limits work – I’ve got an Incus VM running on trixie right now configured with 8GiB RAM.

If possible, could you try using Incus 6.0.5-2 from unstable and see if you still get the same issue? (6.0.5-1 had a bug where you couldn’t incus shell into a VM, so don’t use that one.)

If we’re able to pinpoint the fix, I can work on preparing an update for trixie’s Incus packaging.

3

Thanks for getting back to me so quickly @gibmat :slight_smile:

I’ve upgraded the package to 6.0.5-2, but get the same error. Which version of Hyper-V and CPU are you using? Maybe that makes a difference?

4

For a friend of mine, the reproducer also hit the same barrier on an AMD Ryzen 5 5500U based system. 2816*1024*1024-1 byte worked for him, while 2816*1024*1024 bytes didn’t, so this seems to be the actual barrier.

5

I don’t know if HyperV is doing anything exciting with how it sets up the memory mapping for its layer of the VM.

I have some dev clusters here that run nested but Incus+QEMU all the way and the parent VM has 64GB of RAM with child VM running with 16GB without hitting any issue.

6

Thank you very much for getting back to me, good to hear that this is supported in pure QEMU environments. Is running under Hyper-V generally an unsupported configuration?

Is there a good way to get the commands used by Incus under the hood to launch QEMU? Maybe it’s a general problem unrelated to Incus, and I would then open an issue with the upstream.

Alternatively, I could also try installing Windows on another machine with an Intel 10th (?) Gen CPU, to ensure it’s not a Hyper-V-on-AMD issue, since they had introduced support for nested virtualization much more recently than for Intel iirc.

Do you have other recommendations for me on how to investigate this further?

7

I’ve now tried this on an Intel(R) Core(TM) i5-10500T, and the VM starts without error.

Could this be an AMD + Hyper-V specific thing?

8

I’ve now tried openSUSE Tumbleweed on the AMD Ryzen 7 3800X + Hyper-V system to compare with a rolling-release distribution, and it seems to work.

Installation all defaults, only server as profile.

zypper dup
zypper install incus
systemctl start incus
# all defaults
incus admin init
incus init images:debian/13 debian-test --vm -c limits.memory=8GB
incus start debian-test --console
# boots to the debian login prompt

openSUSE package versions:

incus-6.15-1.1.x86_64
qemu-10.1.0-2.1.x86_64
kernel-default-6.17.0-2.1.x86_64

I guess I’ll have to compare package versions next to attempt to narrow it down, and will also try on NixOS with a recent kernel, as that’s the distribution I originally planned to use Incus on.

9

I tried running a nested VM directly with just QEMU on the Trixie VM like this:

qemu-system-x86_64 -nographic -m 8G -hda debian-img.qcow

This seems to work fine, so my current assumption is that the crash may be caused by some of the extra devices that Incus uses or the interactions with them, but debugging that is difficult.

When I tried grabbing the commandline arguments from the spawned process and removing the fd based devices from the config, it seems to not crash (although also I didn’t get a console).

I then built Incus 6.17 from source in the VM to see whether the crash was fixed in a newer release, but the crash still happens.

I’m not really sure how to proceed here.

@stgraber do you have pointers for me on how to approach debugging this further, and possibly an estimation how difficult this will likely become? If it’s likely to lead to no success, it would also be an option for me to give up on Hyper-V + nesting and use a real test system, although it will be more inconvenient.

10

It’s almost certainly some interaction with HyperV.

Incus assumes most of the VM through QMP, using CPU hotplug, memory hotplug and PCI hotplug to put the whole config together.

So you can’t really just take the Incus QEMU command and run it by hand as that will have maybe 10% of the actual configuration.

Your best bet is to get to run Incus from source so you can modify the logic by hand, but if the issue is with how Hyper-V does things, we’re not very likely to change Incus’ behavior just for that.

11

@stgraber Thank you very much for getting back to me :slight_smile:

Compiling from source on Debian, and experimenting with commenting out devices etc unfortunately didn’t lead me to much.

I’ve then tried running a nested VM like above on Arch Linux and Ubuntu 25.10 and it also failed, giving me a kvm_amd error like on NixOS and Debian, hitting the same assertion.

Since Arch is also a rolling release distribution with the latest packages, openSUSE having newer packages doesn’t seem to be the cause why it runs there.

I then looked into whether something is special about openSUSE Tumbleweed, and it seems like openSUSE is maintaining their own QEMU repository with some extra patches applied (difference to qemu repo).

Unfortunately I didn’t get vanilla QEMU to compile and install properly on openSUSE, Incus complained about missing Firmware, so I can’t confirm whether that’s the actual reason.

Would you recommend opening an issue with QEMU for this?