RHEL based VMs no longer booting, Exception on boot

I have a problem with newer Alma and Rocky Linux VM images, the don’t boot on my system. On the console I see the following output:

  Booting `Rocky Linux (5.14.0-503.26.1.el9_5.x86_64) 9.5 (Blue Onyx)'
 
!!!! X64 Exception Type - 0E(#PF - Page-Fault)  CPU Apic ID - 00000000 !!!!
ExceptionData - 0000000000000003  I:0 R:0 U:0 W:1 P:1 PK:0 SS:0 SGX:0
RIP  - 000000006F091C30, CS  - 0000000000000038, RFLAGS - 0000000000210006
RAX  - 000000006F0AA000, RCX - 000000006F0AA000, RDX - 0000000000024000
RBX  - 0000000076B84500, RSP - 0000000078F9AAD8, RBP - 0000000076D0FE18
RSI  - 0000000000000000, RDI - 000000006F0AA000
R8   - 000000006F0CE000, R9  - 0000000074D8B6FD, R10 - 0000000076609DB9
R11  - 0000000000000077, R12 - 00000000789EC018, R13 - 000000007744B000
R14  - 0000000076B83768, R15 - 0000000076B83770
DS   - 0000000000000030, ES  - 0000000000000030, FS  - 0000000000000030
GS   - 0000000000000030, SS  - 0000000000000030
CR0  - 0000000080010033, CR2 - 000000006F0AA000, CR3 - 0000000078C01000
CR4  - 0000000000000668, CR8 - 0000000000000000
DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
GDTR - 00000000789E1000 0000000000000047, LDTR - 0000000000000000
IDTR - 0000000078351018 0000000000000FFF,   TR - 0000000000000000
FXSAVE_STATE - 0000000078F9A730
!!!! Find image based on IP(0x6F091C30) (No PDB)  (ImageBase=0000000000F61E5C, EntryPoint=0000000000F62BEE) !!!!

A manually downloaded and imported Rocky Linux image works (built timestamp on 2025-02-25), this affects the images 11408f583fdb and 581a25e78492 for me.

Unfortunately I wrote too soon, the problem still exists for me. Steps to reproduce:

incus create --vm images:almalinux/9/cloud almatest
incus config device add almatest agentconfig disk source=agent:config
incus start almatest

To see if it started: incus console --show-log almatest

Edit: This may be related to this grub issue found in SuSE Linux: VMware VM no longer boots after changing to UEFI and specifying GRUB_TERMINAL="serial console" | Support | SUSE

Same here.

Unable to boot Rocky/Alma/RedHat/Fedora vms

  Booting `Fedora Linux (6.13.5-200.fc41.x86_64) 41 (Container Image)'

!!!! X64 Exception Type - 0E(#PF - Page-Fault)  CPU Apic ID - 00000000 !!!!
ExceptionData - 0000000000000003  I:0 R:0 U:0 W:1 P:1 PK:0 SS:0 SGX:0
RIP  - 00000000340A6620, CS  - 0000000000000038, RFLAGS - 0000000000210046
RAX  - 00000000340C2000, RCX - 00000000340C2000, RDX - 0000000000000000
RBX  - 000000003D433618, RSP - 000000003EF3BEE8, RBP - 000000003E9EC018
RSI  - 0000000000000000, RDI - 0000000034104078
R8   - 0000000000000000, R9  - 000000003AD41195, R10 - 000000003C300589
R11  - 0000000000000077, R12 - 000000003CDF5000, R13 - 0000000000000000
R14  - 000000003CB626E8, R15 - 000000003CB626F0
DS   - 0000000000000030, ES  - 0000000000000030, FS  - 0000000000000030
GS   - 0000000000000030, SS  - 0000000000000030
CR0  - 0000000080010033, CR2 - 00000000340C2000, CR3 - 000000003EC01000
CR4  - 0000000000000668, CR8 - 0000000000000000
DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
GDTR - 000000003E9E1000 0000000000000047, LDTR - 0000000000000000
IDTR - 000000003E351018 0000000000000FFF,   TR - 0000000000000000
FXSAVE_STATE - 000000003EF3BB40
!!!! Find image based on IP(0x340A6620) (No PDB)  (ImageBase=0000000000F61E5C, EntryPoint=0000000000F62BEE) !!!!

Regards,

What version of incus are you running? What OS?

I have an similar situation with archlinux VM, with a brief inspection it looks that INCUS KVM does not recognise a hard drive anymore and is unable to mount a root filesystem. I am digging it deeper right now.

Incus 1:6.10-debian12-202502281818, storage on ZFS

EDIT: OK, my issue is related to IOMMU changes.

I have recently upgraded to incus 6.10, and there are new images available on the images: server which work for me.

This topic has a workaround which might help with these issues, they seem to be related to the IOMMU change in 6.10

Hi,

Im using Linux Mint 22.1 (Xia).

Incus installed from zabbly stable repo; current package versions:

ii  incus                                          1:6.9-ubuntu24.04-202502251728             amd64        Incus - Container and virtualization daemon
ii  incus-base                                     1:6.9-ubuntu24.04-202502251728             amd64        Incus - Container and virtualization daemon (container-only)
ii  incus-client                                   1:6.9-ubuntu24.04-202502251728             amd64        Incus - Command line client
ii  incus-extra                                    1:6.9-ubuntu24.04-202502251728             amd64        Incus - Extra tools
ii  incus-ui-canonical                             1:6.9-ubuntu24.04-202502251728             amd64        Incus - Web interface (canonical)

I have a pending update of incus packages. I’ve updated:

ii  incus                                          1:6.10-ubuntu24.04-202503011009            amd64        Incus - Container and virtualization daemon
ii  incus-base                                     1:6.10-ubuntu24.04-202503011009            amd64        Incus - Container and virtualization daemon (container-only)
ii  incus-client                                   1:6.10-ubuntu24.04-202503011009            amd64        Incus - Command line client
ii  incus-extra                                    1:6.10-ubuntu24.04-202503011009            amd64        Incus - Extra tools
ii  incus-ui-canonical                             1:6.10-ubuntu24.04-202503011009            amd64        Incus - Web interface (canonical)

And now the vms starts without problem.

Regards

What QEMU versions are you all using?

I faced the same issue with incus 6.7 im my test environment (Ubuntu server under Hyper-V. It gone away after updating to incus 6.9 and latest qemu* packages.

P.S.
Pay attention if you plan to upgrade to Incus 6.10. Changes in VM support may impact on newly deployed VMs: IOMMU emulation sometimes triggers kernel boot issue

What QEMU versions are you all using?

For me incus seems to be using its own QEMU in /opt/incus/bin/qemu-system-x86_64, unfortunately since I upgraded incus I don’t know which version was shipped with 6.9.

I don’t think the problem for Alma or Rocky was related to IOMMU, it looked to me like it was grub that failed and the kernel wasn’t even loaded. I don’t know if the problem was fixed in new images or due to incus/qemu upgrade.