RHEL based VMs no longer booting, Exception on boot

Incus
1

I have a problem with newer Alma and Rocky Linux VM images, the don’t boot on my system. On the console I see the following output:

  Booting `Rocky Linux (5.14.0-503.26.1.el9_5.x86_64) 9.5 (Blue Onyx)'
 
!!!! X64 Exception Type - 0E(#PF - Page-Fault)  CPU Apic ID - 00000000 !!!!
ExceptionData - 0000000000000003  I:0 R:0 U:0 W:1 P:1 PK:0 SS:0 SGX:0
RIP  - 000000006F091C30, CS  - 0000000000000038, RFLAGS - 0000000000210006
RAX  - 000000006F0AA000, RCX - 000000006F0AA000, RDX - 0000000000024000
RBX  - 0000000076B84500, RSP - 0000000078F9AAD8, RBP - 0000000076D0FE18
RSI  - 0000000000000000, RDI - 000000006F0AA000
R8   - 000000006F0CE000, R9  - 0000000074D8B6FD, R10 - 0000000076609DB9
R11  - 0000000000000077, R12 - 00000000789EC018, R13 - 000000007744B000
R14  - 0000000076B83768, R15 - 0000000076B83770
DS   - 0000000000000030, ES  - 0000000000000030, FS  - 0000000000000030
GS   - 0000000000000030, SS  - 0000000000000030
CR0  - 0000000080010033, CR2 - 000000006F0AA000, CR3 - 0000000078C01000
CR4  - 0000000000000668, CR8 - 0000000000000000
DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
GDTR - 00000000789E1000 0000000000000047, LDTR - 0000000000000000
IDTR - 0000000078351018 0000000000000FFF,   TR - 0000000000000000
FXSAVE_STATE - 0000000078F9A730
!!!! Find image based on IP(0x6F091C30) (No PDB)  (ImageBase=0000000000F61E5C, EntryPoint=0000000000F62BEE) !!!!

A manually downloaded and imported Rocky Linux image works (built timestamp on 2025-02-25), this affects the images 11408f583fdb and 581a25e78492 for me.

3

Unfortunately I wrote too soon, the problem still exists for me. Steps to reproduce:

incus create --vm images:almalinux/9/cloud almatest
incus config device add almatest agentconfig disk source=agent:config
incus start almatest

To see if it started: incus console --show-log almatest

Edit: This may be related to this grub issue found in SuSE Linux: VMware VM no longer boots after changing to UEFI and specifying GRUB_TERMINAL="serial console" | Support | SUSE

4

Same here.

Unable to boot Rocky/Alma/RedHat/Fedora vms

  Booting `Fedora Linux (6.13.5-200.fc41.x86_64) 41 (Container Image)'

!!!! X64 Exception Type - 0E(#PF - Page-Fault)  CPU Apic ID - 00000000 !!!!
ExceptionData - 0000000000000003  I:0 R:0 U:0 W:1 P:1 PK:0 SS:0 SGX:0
RIP  - 00000000340A6620, CS  - 0000000000000038, RFLAGS - 0000000000210046
RAX  - 00000000340C2000, RCX - 00000000340C2000, RDX - 0000000000000000
RBX  - 000000003D433618, RSP - 000000003EF3BEE8, RBP - 000000003E9EC018
RSI  - 0000000000000000, RDI - 0000000034104078
R8   - 0000000000000000, R9  - 000000003AD41195, R10 - 000000003C300589
R11  - 0000000000000077, R12 - 000000003CDF5000, R13 - 0000000000000000
R14  - 000000003CB626E8, R15 - 000000003CB626F0
DS   - 0000000000000030, ES  - 0000000000000030, FS  - 0000000000000030
GS   - 0000000000000030, SS  - 0000000000000030
CR0  - 0000000080010033, CR2 - 00000000340C2000, CR3 - 000000003EC01000
CR4  - 0000000000000668, CR8 - 0000000000000000
DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
GDTR - 000000003E9E1000 0000000000000047, LDTR - 0000000000000000
IDTR - 000000003E351018 0000000000000FFF,   TR - 0000000000000000
FXSAVE_STATE - 000000003EF3BB40
!!!! Find image based on IP(0x340A6620) (No PDB)  (ImageBase=0000000000F61E5C, EntryPoint=0000000000F62BEE) !!!!

Regards,

5

What version of incus are you running? What OS?

6

I have an similar situation with archlinux VM, with a brief inspection it looks that INCUS KVM does not recognise a hard drive anymore and is unable to mount a root filesystem. I am digging it deeper right now.

Incus 1:6.10-debian12-202502281818, storage on ZFS

EDIT: OK, my issue is related to IOMMU changes.

7

I have recently upgraded to incus 6.10, and there are new images available on the images: server which work for me.

8

This topic has a workaround which might help with these issues, they seem to be related to the IOMMU change in 6.10

9

Hi,

Im using Linux Mint 22.1 (Xia).

Incus installed from zabbly stable repo; current package versions:

ii  incus                                          1:6.9-ubuntu24.04-202502251728             amd64        Incus - Container and virtualization daemon
ii  incus-base                                     1:6.9-ubuntu24.04-202502251728             amd64        Incus - Container and virtualization daemon (container-only)
ii  incus-client                                   1:6.9-ubuntu24.04-202502251728             amd64        Incus - Command line client
ii  incus-extra                                    1:6.9-ubuntu24.04-202502251728             amd64        Incus - Extra tools
ii  incus-ui-canonical                             1:6.9-ubuntu24.04-202502251728             amd64        Incus - Web interface (canonical)

I have a pending update of incus packages. I’ve updated:

ii  incus                                          1:6.10-ubuntu24.04-202503011009            amd64        Incus - Container and virtualization daemon
ii  incus-base                                     1:6.10-ubuntu24.04-202503011009            amd64        Incus - Container and virtualization daemon (container-only)
ii  incus-client                                   1:6.10-ubuntu24.04-202503011009            amd64        Incus - Command line client
ii  incus-extra                                    1:6.10-ubuntu24.04-202503011009            amd64        Incus - Extra tools
ii  incus-ui-canonical                             1:6.10-ubuntu24.04-202503011009            amd64        Incus - Web interface (canonical)

And now the vms starts without problem.

Regards

10

What QEMU versions are you all using?

11

I faced the same issue with incus 6.7 im my test environment (Ubuntu server under Hyper-V. It gone away after updating to incus 6.9 and latest qemu* packages.

P.S.
Pay attention if you plan to upgrade to Incus 6.10. Changes in VM support may impact on newly deployed VMs: IOMMU emulation sometimes triggers kernel boot issue

12

What QEMU versions are you all using?

For me incus seems to be using its own QEMU in /opt/incus/bin/qemu-system-x86_64, unfortunately since I upgraded incus I don’t know which version was shipped with 6.9.

I don’t think the problem for Alma or Rocky was related to IOMMU, it looked to me like it was grub that failed and the kernel wasn’t even loaded. I don’t know if the problem was fixed in new images or due to incus/qemu upgrade.