I’m trying to create a VM using LXD. The information on the server I am working on is as follows.
os: Linux cpu01 5.4.0-165-generic #182-Ubuntu SMP Mon Oct 2 19:43:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
When I issue the lxd command as follows, the result is as follows.
user:~$ lxc launch ubuntu:20.04 testVm -c limits.cpu=64 --vm lxc info --show-log local:testVm for more info
When the log is output, it says something like this.
user:~$ lxc info --show-log testVm
What should I do in these cases?
stgraber
November 10, 2023, 5:45pm
2
Can you show lscpu from your host machine?
I’m sorry for late reply. And I think some of the facts I knew were wrong and need to be corrected.lscpu output result.
Architecture: x86_64
stgraber
November 14, 2023, 5:07pm
4
Very interesting. Based on the error, I think we should change our number of max hot-plugable CPUs to 240 in Incus, which will then fix this issue.
stgraber
November 14, 2023, 5:31pm
5
So I talked to a friend who works at AMD and he confirmed this was a QEMU bug but that this has been resolved.
I’m going to test it now using the latest Incus packages to see what happens on current QEMU.
stgraber
November 14, 2023, 5:49pm
6
Can you show lxc info on the affected system?
I’ve confirmed that newer QEMU has bumped the limit to 1024, so it’s odd that this is still happening.
stgraber
November 14, 2023, 5:50pm
7
It looks like QEMU 8.1.x should be fine, QEMU 8.0.x however still has the 288 limit.
stgraber
November 15, 2023, 4:54am
9
That’s odd, QEMU 8.1.1 should have the much higher limit in place…
stgraber
November 18, 2023, 10:52pm
10
As I don’t own a dual-socket EPYC with an insane number of cores, I’ve had to simulate it using QEMU:
root@ubuntu:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 384
On-line CPU(s) list: 0-383
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 7 5700G with Radeon Graphics
CPU family: 25
Model: 80
Thread(s) per core: 2
Core(s) per socket: 96
Socket(s): 2
Stepping: 0
BogoMIPS: 7599.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mc
a cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall n
x mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid
extd_apicid amd_dcm tsc_known_freq pni pclmulqdq ssse3
fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadlin
e_timer aes xsave avx f16c rdrand hypervisor lahf_lm cm
p_legacy svm cr8_legacy abm sse4a misalignsse 3dnowpref
etch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmm
call fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpc
id rdseed adx smap clflushopt clwb sha_ni xsaveopt xsav
ec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt l
brv nrip_save tsc_scale vmcb_clean pausefilter pfthresh
old v_vmsave_vmload vgif umip pku ospke vaes vpclmulqdq
rdpid fsrm arch_capabilities
Virtualization features:
Virtualization: AMD-V
Hypervisor vendor: KVM
Virtualization type: full
Caches (sum of all):
L1d: 12 MiB (192 instances)
L1i: 12 MiB (192 instances)
L2: 96 MiB (192 instances)
L3: 256 MiB (16 instances)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-383
Vulnerabilities:
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Retbleed: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer
sanitization
Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIB
P always-on, RSB filling, PBRSB-eIBRS Not affected
Srbds: Not affected
Tsx async abort: Not affected
I’ve then asked Incus to create me a VM similar to what you used for LXD:
root@ubuntu:~# incus launch images:ubuntu/22.04 v1 -c limits.cpu=64 -c limits.memory=4GiB --vm
Creating v1
Starting v1
root@ubuntu:~# incus exec v1 -- lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0
Off-line CPU(s) list: 1-63
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 7 5700G with Radeon Graphics
CPU family: 25
Model: 80
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 1
Stepping: 0
BogoMIPS: 7599.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mc
a cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall n
x mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid
extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx1
6 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer
aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy
svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osv
w perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase ts
c_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx sm
ap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsave
s clzero xsaveerptr wbnoinvd arat npt lbrv nrip_save ts
c_scale vmcb_clean pausefilter pfthreshold v_vmsave_vml
oad vgif umip pku ospke vaes vpclmulqdq rdpid fsrm arch
_capabilities
Virtualization features:
Virtualization: AMD-V
Caches (sum of all):
L1d: 64 KiB (1 instance)
L1i: 64 KiB (1 instance)
L2: 512 KiB (1 instance)
L3: 16 MiB (1 instance)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Retbleed: Not affected
Spec rstack overflow: Mitigation; safe RET, no microcode
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
and seccomp
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer
sanitization
Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIB
P disabled, RSB filling, PBRSB-eIBRS Not affected
Srbds: Not affected
Tsx async abort: Not affected
root@ubuntu:~#
I’ve also confirmed that it’s possible to create a VM on Incus with more CPUs then the old limit:
root@ubuntu:~# incus launch images:ubuntu/22.04 v2 -c limits.cpu=256 -c limits.memory=4GiB --vm
Creating v2
Starting v2
So the newer version of QEMU we ship with Incus seems to be behaving just fine with those larger systems. I’m unsure why the one you’ve got on your system is somehow failing to handle this.