AlmaLinux and RockyLinux VMs panicking

Incus
1

See AlmaLinux and RockyLinux VM panicking. · Issue #3604 · lxc/incus · GitHub for info. Stephane suggested I post here for additional insight.

2

Would be useful if you could provide more details on the system itself, especially CPU vendor and model, QEMU version, kernel log output, …

3

This is a very old document, but still very relevant today: The SIG11 problem

Kernel builds are good at stressing the hardware, and so are running VMs.

I’ve been bitten by this myself in the past. I was able to prove it was a bad DIMM on a 2-DIMM system by running with just one DIMM, and then with just the other.

4

We need to wait for more details to know, but something else that comes to mind would be some of the new Intel consumer grade chips with a mix of cores. Not all cores may support the same features, so the VM could get re-scheduled over to a different core and then hit some odd issues when specific CPU extensions just disappear.

The kernel should be reasonably smart to avoid that, but it may still be an issue.

Anyway, hard to tell until we know more about the system.

5 
adavis@boss5 (- /1 -) [~] $ lscpu
Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             42 bits physical, 48 bits virtual
  Byte Order:                Little Endian
CPU(s):                      16
  On-line CPU(s) list:       0-15
Vendor ID:                   GenuineIntel
  Model name:                Intel(R) Core(TM) Ultra 7 255H
    CPU family:              6
    Model:                   197
    Thread(s) per core:      1
    Core(s) per socket:      16
    Socket(s):               1
    Stepping:                2
    Microcode version:       0x121
    Frequency boost:         enabled
    CPU(s) scaling MHz:      57%
    CPU max MHz:             2001.0000
    CPU min MHz:             400.0000
    BogoMIPS:                7372.80
    Flags:                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1
                             gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dt
                             es64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave av
                             x f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsb
                             ase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_loc
                             k_detect avx_vnni lam wbnoinvd dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg
                              gfni vaes vpclmulqdq rdpid bus_lock_detect movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities
Virtualization features:     
  Virtualization:            VT-x
Caches (sum of all):         
  L1d:                       480 KiB (12 instances)
  L1i:                       768 KiB (12 instances)
  L2:                        28 MiB (9 instances)
  L3:                        24 MiB (1 instance)
NUMA:                        
  NUMA node(s):              1
  NUMA node0 CPU(s):         0-15
Vulnerabilities:             
  Gather data sampling:      Not affected
  Ghostwrite:                Not affected
  Indirect target selection: Not affected
  Itlb multihit:             Not affected
  L1tf:                      Not affected
  Mds:                       Not affected
  Meltdown:                  Not affected
  Mmio stale data:           Not affected
  Old microcode:             Not affected
  Reg file data sampling:    Not affected
  Retbleed:                  Not affected
  Spec rstack overflow:      Not affected
  Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:                Vulnerable: eIBRS with unprivileged eBPF
  Srbds:                     Not affected
  Tsa:                       Not affected
  Tsx async abort:           Not affected
  Vmscape:                   Mitigation; IBPB before exit to userspace


Linux boss5.example.org 6.18.38 #1 SMP PREEMPT_DYNAMIC Sat Jul  4 19:11:50 EDT 2026 sx86_64 Intel(R) Core(TM) Ultra 7 255H GenuineIntel GNU/Linux

QEMU version: 10.0.11

64GB RAM. Nothing in the kernel logs.

6

Okay, so it is a setup with 3 tiers of cores.
You may want to use limits.cpu to pin the VM on just one set of cores to see if that helps.

7

Weird thing: this issue doesn’t happen with a Debian VM.

8

Yeah, it probably has some interaction with the guest kernel given that both Alma and Rocky would be on a similar kernel version and config.

9

I ran the following commands:

incus profile set vm-pcore limits.cpu='0-3'
incus launch images:almalinux/10 vmtest-alma -p agent -p vm-pcore --vm

We’ll see how it goes.

10

After about 2 hours, it panicked. I ran a memory check; that passed. I’ll run a VM in virt-manager to see if it happens there.

@stgraber When you test VMs, how many vCPUS do you allocate?

11

The automated image validation runs with just 1 CPU and 1GiB of RAM on 10GiB of disk (Incus defaults).

The Incus pre-release tests use long running VMs and those are more beefy, typically 4 vCPU and 8GB of RAM.