VM not booting/installing from Windows 11 multi-edition ISO

Libvirt/QEMU UEFI works fine on a 5.10.179-gentoo-dist kernel, and the portage version 5.0.2-r2’s LXD containers work for the most part other than a stubborn nvenc plex issue - for another time. The headless & unpriveleged LXD steam streaming container works very well, too. The distrobuilder process to repack-windows completes without error for the Windows 11 multi-edition ISO for x64 devices downloaded from https://www.microsoft.com/en-us/software-download/windows11

However, following Stephane’s instruction to include adjusted needs at https://www.youtube.com/watch?v=3PDMGwbbk48 as:
lxc init win11 --vm --empty
lxc config set win11 limits.cpu=8 limits.memory=32GiB
lxc config device override win11 root size=100GiB
lxc config device add win11 install disk source=/nfs/ISO/lxd_repack-Win11_22H2_English_x64v1.iso boot.priority=10
lxc start win11 --console=vga

Pressing a key to start the Windows installation seems to hang the session with a single CPU holding 1 core at 100%. Escaping to the bootloader shows fs0:\efi\boot\bootx64.efi This is the same for the original ISO - not repacked.

I may I have read every post and guidance for LXD4 and LXD5 google has yielded. At this point I can’t seem to identify additional areas to continue to troubleshoot. Therefore, debugging hints or other troubleshooting guidance is greatly appreciated.

It seems there is a kernel bug triggered when using LXD/QEMU to boot Win11. No bug with Libvirt/QEMU

DMESG output:
[Thu Jun 1 15:53:36 2023] BUG: kernel NULL pointer dereference, address: 0000000000000000
[Thu Jun 1 15:53:36 2023] #PF: supervisor read access in kernel mode
[Thu Jun 1 15:53:36 2023] #PF: error_code(0x0000) - not-present page
[Thu Jun 1 15:53:36 2023] PGD 0 P4D 0
[Thu Jun 1 15:53:36 2023] Oops: 0000 [#17] SMP PTI
[Thu Jun 1 15:53:36 2023] CPU: 0 PID: 17965 Comm: qemu-system-x86 Tainted: P D O 5.10.179-gentoo-dist #1
[Thu Jun 1 15:53:36 2023] Hardware name: SuperMicro/To be filled by O.E.M., BIOS 5.6.5 06/18/2015
[Thu Jun 1 15:53:36 2023] RIP: 0010:find_first_bit+0x19/0x40
[Thu Jun 1 15:53:36 2023] Code: 5d 41 5e 41 5f c3 cc cc cc cc cc cc cc cc cc cc cc 48 85 f6 48 89 f0 74 2d 31 d2 eb 0d 48 83 c2 40 48 83 c7 08 48 39 c2 73 1c <48> 8b 0f 48 85 c9 74 eb f3 48 0f bc c9 48 01 d1 48 39 c8 48 0f 47
[Thu Jun 1 15:53:36 2023] RSP: 0018:ffffb33accd9fa68 EFLAGS: 00010246
[Thu Jun 1 15:53:36 2023] RAX: 0000000000000120 RBX: ffffb33b025a2000 RCX: 0000000000000000
[Thu Jun 1 15:53:36 2023] RDX: 0000000000000000 RSI: 0000000000000120 RDI: 0000000000000000
[Thu Jun 1 15:53:36 2023] RBP: 0000000000000000 R08: ffff962b13229ff8 R09: ffff962b13229ff8
[Thu Jun 1 15:53:36 2023] R10: 000000000000000e R11: 0000000000000420 R12: ffff962b13229ff8
[Thu Jun 1 15:53:36 2023] R13: 0000000000000323 R14: 0000000000000000 R15: 0000000000000000
[Thu Jun 1 15:53:36 2023] FS: 00007fda8e88d6c0(0000) GS:ffff9627df800000(0000) knlGS:fffff8027242a000
[Thu Jun 1 15:53:36 2023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Thu Jun 1 15:53:36 2023] CR2: 0000000000000000 CR3: 0000002af2a0c001 CR4: 00000000001726f0
[Thu Jun 1 15:53:36 2023] Call Trace:
[Thu Jun 1 15:53:36 2023] kvm_make_vcpus_request_mask+0x3a/0xf0 [kvm]
[Thu Jun 1 15:53:36 2023] kvm_hv_get_assist_page+0x8f8/0xb10 [kvm]
[Thu Jun 1 15:53:36 2023] ? kvm_configure_mmu+0xd41/0x2560 [kvm]
[Thu Jun 1 15:53:36 2023] ? kvm_configure_mmu+0x1f6e/0x2560 [kvm]
[Thu Jun 1 15:53:36 2023] ? kvm_init_shadow_npt_mmu+0x1318/0x22c0 [kvm]
[Thu Jun 1 15:53:36 2023] ? kvm_init_shadow_npt_mmu+0x168b/0x22c0 [kvm]
[Thu Jun 1 15:53:36 2023] ? kvm_is_reserved_pfn+0xe/0x420 [kvm]
[Thu Jun 1 15:53:36 2023] ? kvm_release_pfn_clean+0x22/0x40 [kvm]
[Thu Jun 1 15:53:36 2023] ? disallowed_hugepage_adjust+0x25b/0xab0 [kvm]
[Thu Jun 1 15:53:36 2023] ? kvm_mmu_page_fault+0x67/0x600 [kvm]
[Thu Jun 1 15:53:36 2023] ? kvm_lapic_reg_write+0xe8/0x670 [kvm]
[Thu Jun 1 15:53:36 2023] ? kvm_emulate_hypercall+0x25/0x540 [kvm]
[Thu Jun 1 15:53:36 2023] kvm_hv_hypercall+0x154/0x4d0 [kvm]
[Thu Jun 1 15:53:36 2023] ? pi_update_irte+0x200f/0x20a0 [kvm_intel]
[Thu Jun 1 15:53:36 2023] kvm_arch_vcpu_ioctl_run+0x657/0x15d0 [kvm]
[Thu Jun 1 15:53:36 2023] ? do_futex+0x47e/0xb40
[Thu Jun 1 15:53:36 2023] kvm_vcpu_block+0x510/0x9a0 [kvm]
[Thu Jun 1 15:53:36 2023] ? __fget_files+0x76/0xa0
[Thu Jun 1 15:53:36 2023] __x64_sys_ioctl+0x90/0xd0
[Thu Jun 1 15:53:36 2023] do_syscall_64+0x33/0x80
[Thu Jun 1 15:53:36 2023] entry_SYSCALL_64_after_hwframe+0x61/0xc6
[Thu Jun 1 15:53:36 2023] RIP: 0033:0x7fda909bdc7b
[Thu Jun 1 15:53:36 2023] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
[Thu Jun 1 15:53:36 2023] RSP: 002b:00007fda8e88c6a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Thu Jun 1 15:53:36 2023] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007fda909bdc7b
[Thu Jun 1 15:53:36 2023] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 000000000000001c
[Thu Jun 1 15:53:36 2023] RBP: 000055d84b3643f0 R08: 000055d849c97990 R09: 00000000ffffffff
[Thu Jun 1 15:53:36 2023] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Thu Jun 1 15:53:36 2023] R13: 0000000000000007 R14: 00007fff42678ed0 R15: 00007fda8e08d000
[Thu Jun 1 15:53:37 2023] Modules linked in: isofs overlay udf crc_itu_t loop vhost_net macvtap macvlan tap tun nfsv3 nfs_acl cfg80211 rfkill veth nf_conntrack_netlink xt_addrtype br_netfilter bridge ebtable_filter ebtables ip6table_raw ip6table_mangle ip6table_nat ip6table_filter ip6_tables iptable_raw iptable_mangle iptable_nat iptable_filter ip_tables vhost_vsock vmw_vsock_virtio_transport_common vhost vhost_iotlb vsock fuse rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc nfs_ssc fscache nfnetlink openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 8021q garp mrp stp llc zram binfmt_misc intel_rapl_msr snd_hda_codec_hdmi intel_rapl_common sb_edac snd_hda_intel x86_pkg_temp_thermal snd_intel_dspcfg intel_powerclamp soundwire_intel soundwire_generic_allocation snd_soc_core kvm_intel snd_compress snd_pcm_dmaengine soundwire_cadence kvm snd_hda_codec nct6775 snd_hda_core hwmon_vid ac97_bus snd_hwdep irqbypass snd_pcm iTCO_wdt rapl snd_timer intel_pmc_bxt
[Thu Jun 1 15:53:37 2023] lm75 iTCO_vendor_support intel_cstate i2c_i801 snd soundcore intel_uncore coretemp pcspkr i2c_smbus lpc_ich zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) crct10dif_pclmul crc32_pclmul crc32c_intel spl(O) ghash_clmulni_intel mpt3sas igb nvme raid_class i2c_algo_bit scsi_transport_sas dca nvme_core wmi nvidia_drm(PO) drm_kms_helper cec nvidia_uvm(PO) nvidia_modeset(PO) nvidia(PO) drm
[Thu Jun 1 15:53:37 2023] CR2: 0000000000000000
[Thu Jun 1 15:53:37 2023] —[ end trace 15c1706e8c156914 ]—
[Thu Jun 1 15:53:37 2023] RIP: 0010:find_first_bit+0x19/0x40
[Thu Jun 1 15:53:37 2023] Code: 5d 41 5e 41 5f c3 cc cc cc cc cc cc cc cc cc cc cc 48 85 f6 48 89 f0 74 2d 31 d2 eb 0d 48 83 c2 40 48 83 c7 08 48 39 c2 73 1c <48> 8b 0f 48 85 c9 74 eb f3 48 0f bc c9 48 01 d1 48 39 c8 48 0f 47
[Thu Jun 1 15:53:37 2023] RSP: 0018:ffffb33acc9d7a68 EFLAGS: 00010246
[Thu Jun 1 15:53:37 2023] RAX: 0000000000000120 RBX: ffffb33acd231000 RCX: 0000000000000000
[Thu Jun 1 15:53:37 2023] RDX: 0000000000000000 RSI: 0000000000000120 RDI: 0000000000000000
[Thu Jun 1 15:53:37 2023] RBP: 0000000000000000 R08: ffff962919bdcff8 R09: ffff962919bdcff8
[Thu Jun 1 15:53:37 2023] R10: 000000000000000e R11: 0000000000000000 R12: ffff962919bdcff8
[Thu Jun 1 15:53:37 2023] R13: 0000000000000323 R14: 0000000000000000 R15: 0000000000000000
[Thu Jun 1 15:53:37 2023] FS: 00007fda8e88d6c0(0000) GS:ffff9627df800000(0000) knlGS:fffff8027242a000
[Thu Jun 1 15:53:37 2023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Thu Jun 1 15:53:37 2023] CR2: 0000000000000000 CR3: 0000002af2a0c001 CR4: 00000000001726f0

LXD and libvirt enables different QEMU flags which might be why you are hitting that kernel/kvm issue.

Run FreeBSD 13.1 / OPNsense 22.7 / pfSense 2.7.0 (and newer?) under LXD VM contains some ideas on things to try. I’d start by trying the -cpu host trick first and see if that fares better.

I do appreciate the comment, and I was hoping there was some hidden trove of documents which would explain in end-user’s tongue the finer point of the default QEMU workflow/config. Different - yes - but specifically what ?

Since this topic seems more of a black box and I am not married to 5.10, I opted to revert to kernel 5.4.242 which doesn’t seem to have said issue…

If you want to compare how QEMU is invoked, you can peak at LXD’s config by inspecting the args passed to QEMU with ps aux | grep qemu and also looking at the QEMU config file that lives at /var/snap/lxd/common/lxd/logs/<NAME>/qemu.conf.

We see that hv_passthrough is used for the -cpu which is one key difference from libvirt’s default:

/snap/lxd/24918/bin/qemu-system-x86_64 -S -name juju-lxd -uuid 83a8be7b-f64f-4537-9a3f-8d1af7edaf1c -daemonize -cpu host,hv_passthrough -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=allow,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/juju-lxd/qemu.conf -spice unix=on,disable-ticketing=on,addr=/var/snap/lxd/common/lxd/logs/juju-lxd/qemu.spice -pidfile /var/snap/lxd/common/lxd/logs/juju-lxd/qemu.pid -D /var/snap/lxd/common/lxd/logs/juju-lxd/qemu.log -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd

And LXD uses a modern machine q35 model and uses UEFI, both also not what libvirt does by default:

$ sudo cat /var/snap/lxd/common/lxd/logs/juju-lxd/qemu.conf
# Machine
[machine]
graphics = "off"
type = "q35"
accel = "kvm"
usb = "off"

[global]
driver = "ICH9-LPC"
property = "disable_s3"
value = "1"

[global]
driver = "ICH9-LPC"
property = "disable_s4"
value = "1"

[boot-opts]
strict = "on"

# Memory
[memory]
size = "10240M"

# CPU
[smp-opts]
cpus = "1"
maxcpus = "12"

[object "mem0"]
qom-type = "memory-backend-file"
mem-path = "/dev/hugepages"
prealloc = "on"
discard-data = "on"
size = "10240M"
share = "on"

[numa]
type = "node"
nodeid = "0"
memdev = "mem0"

# Firmware (read only)
[drive]
file = "/snap/lxd/current/share/qemu/OVMF_CODE.fd"
if = "pflash"
format = "raw"
unit = "0"
readonly = "on"

# Firmware settings (writable)
[drive]
file = "/dev/fd/3"
if = "pflash"
format = "raw"
unit = "1"

# Qemu control
[chardev "monitor"]
backend = "socket"
path = "/var/snap/lxd/common/lxd/logs/juju-lxd/qemu.monitor"
server = "on"
wait = "off"

[mon]
chardev = "monitor"
mode = "control"

# Console
[chardev "console"]
backend = "socket"
path = "/var/snap/lxd/common/lxd/logs/juju-lxd/qemu.console"
server = "on"
wait = "off"

[device "qemu_pcie0"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.0"
chassis = "0"
multifunction = "on"

# Balloon driver
[device "qemu_balloon"]
driver = "virtio-balloon-pci"
bus = "qemu_pcie0"
addr = "00.0"
multifunction = "on"

# Random number generator
[object "qemu_rng"]
qom-type = "rng-random"
filename = "/dev/urandom"

[device "dev-qemu_rng"]
driver = "virtio-rng-pci"
bus = "qemu_pcie0"
addr = "00.1"
rng = "qemu_rng"

# Input
[device "qemu_keyboard"]
driver = "virtio-keyboard-pci"
bus = "qemu_pcie0"
addr = "00.2"

# Input
[device "qemu_tablet"]
driver = "virtio-tablet-pci"
bus = "qemu_pcie0"
addr = "00.3"

# Vsock
[device "qemu_vsock"]
driver = "vhost-vsock-pci"
bus = "qemu_pcie0"
addr = "00.4"
guest-cid = "181"

# Virtual serial bus
[device "dev-qemu_serial"]
driver = "virtio-serial-pci"
bus = "qemu_pcie0"
addr = "00.5"

# LXD serial identifier
[chardev "qemu_serial-chardev"]
backend = "ringbuf"
size = "16B"

[device "qemu_serial"]
driver = "virtserialport"
name = "org.linuxcontainers.lxd"
chardev = "qemu_serial-chardev"
bus = "dev-qemu_serial.0"

# Spice agent
[chardev "qemu_spice-chardev"]
backend = "spicevmc"
name = "vdagent"

[device "qemu_spice"]
driver = "virtserialport"
name = "com.redhat.spice.0"
chardev = "qemu_spice-chardev"
bus = "dev-qemu_serial.0"

# Spice folder
[chardev "qemu_spicedir-chardev"]
backend = "spiceport"
name = "org.spice-space.webdav.0"

[device "qemu_spicedir"]
driver = "virtserialport"
name = "org.spice-space.webdav.0"
chardev = "qemu_spicedir-chardev"
bus = "dev-qemu_serial.0"

# USB controller
[device "qemu_usb"]
driver = "qemu-xhci"
bus = "qemu_pcie0"
addr = "00.6"
p2 = "8"
p3 = "8"

[chardev "qemu_spice-usb-chardev1"]
backend = "spicevmc"
name = "usbredir"

[device "qemu_spice-usb1"]
driver = "usb-redir"
chardev = "qemu_spice-usb-chardev1"

[chardev "qemu_spice-usb-chardev2"]
backend = "spicevmc"
name = "usbredir"

[device "qemu_spice-usb2"]
driver = "usb-redir"
chardev = "qemu_spice-usb-chardev2"

[chardev "qemu_spice-usb-chardev3"]
backend = "spicevmc"
name = "usbredir"

[device "qemu_spice-usb3"]
driver = "usb-redir"
chardev = "qemu_spice-usb-chardev3"

[device "qemu_pcie1"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.1"
chassis = "1"

# SCSI controller
[device "qemu_scsi"]
driver = "virtio-scsi-pci"
bus = "qemu_pcie1"
addr = "00.0"

[device "qemu_pcie2"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.2"
chassis = "2"

# Config drive (9p)
[fsdev "qemu_config"]
fsdriver = "local"
security_model = "none"
readonly = "on"
path = "/var/snap/lxd/common/lxd/devices/juju-lxd/config.mount"

[device "dev-qemu_config-drive-9p"]
driver = "virtio-9p-pci"
bus = "qemu_pcie2"
addr = "00.0"
multifunction = "on"
mount_tag = "config"
fsdev = "qemu_config"

# Config drive (virtio-fs)
[chardev "qemu_config"]
backend = "socket"
path = "/var/snap/lxd/common/lxd/logs/juju-lxd/virtio-fs.config.sock"

[device "dev-qemu_config-drive-virtio-fs"]
driver = "vhost-user-fs-pci"
bus = "qemu_pcie2"
addr = "00.1"
tag = "config"
chardev = "qemu_config"

[device "qemu_pcie3"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.3"
chassis = "3"

# GPU
[device "qemu_gpu"]
driver = "virtio-vga"
bus = "qemu_pcie3"
addr = "00.0"

[device "qemu_pcie4"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.4"
chassis = "4"

# VM Generation ID
[device "vmgenid0"]
driver = "vmgenid"
guid = "83a8be7b-f64f-4537-9a3f-8d1af7edaf1c"

[device "qemu_pcie5"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.5"
chassis = "5"

[device "qemu_pcie6"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.6"
chassis = "6"

[device "qemu_pcie7"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.7"
chassis = "7"

[device "qemu_pcie8"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "2.0"
chassis = "8"
multifunction = "on"

I’m glad this information didn’t head my way prior to downgrading the kernel. I may have been down yet another LXD rabbit hole. It’s useful to know the LXD on gentoo QEMU config is at /var/log/lxd/VM-NAME/qemu.conf - Thank you for highlighting this

5.4.242 will likely remain, and I can start a different topic for the strange plex nvenc container issue which is consistent on both kernels but functions fine in docker…

202306021215-select

I guess the fun doesn’t cease. I thought tpm might be the issue, but it vtpm is in the qemu.conf and populated:
/var/lib/lxd/virtual-machines/VM-Win11x64/tpm.vtpm/swtpm-vtpm.soc

Windows simply does not wish to play well with my install of LXD. Laughter and comment are greatly appreciated.

It maybe to do with this

https://github.com/lxc/lxd/pull/11515

This seems plausible - Thank you for this. However, I seem to be unable to identify an LXD workflow to workaround in either of the LXD discussions or the github discussions. It seems like there was some work performed in 5.14 to address, but is there a workaround workflow I can follow in 5.02 ?

If smbios data is to be trusted and your machine is from 2015, I’d also check if you CPU is compatible with Win11 as they dropped compat with older CPU models.

202306021829-select
I have not observed MS prevents win11 install based on processor. At least these ‘unsupported’ CPUs function fine through Libvirt-based install .

IIRC from long ago, libvirt doesn’t pass the host CPU by default but a different/generic model.

Not certain regarding history and other distros. However, CPU is demonstrably passed-through by default on the system in question.

Although I have gone through the configuration differences a few times, there is nothing which stands-out as the show-stopper preventing Win11 from continuing installation in LXD/QEMU. I even leveraged the exact same lxd repack-windows ISO for both.

In the chance a guru happens to glance at this portion of the thread, I pasted both the Libvirt/QEMU settings for the successful install of Windows 11 multi-edition ISO (Libvirt) as well as the unsuccessful attempted install of Windows 11 multi-edition ISO (LXD). Although, guidance altering the LXD/QEMU portion is appreciated, if a root cause is identified from the config :

Libvirt/QEMU:
starting up libvirt version: 8.9.0, qemu version: 7.2.0, kernel: 5.4.242-gentoo-dist
LC_ALL=C
PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/lib/llvm/15/bin
HOME=/var/lib/libvirt/qemu/domain-12-W11x64
USER=root
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-12-W11x64/.local/share
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-12-W11x64/.cache
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-12-W11x64/.config
/usr/bin/qemu-system-x86_64
-name guest=W11x64,debug-threads=on
-S
-object ‘{“qom-type”:“secret”,“id”:“masterKey0”,“format”:“raw”,“file”:"/var/lib/libvirt/qemu/domain-12-W11x64/master-key.aes"}’
-blockdev ‘{“driver”:“file”,“filename”:"/usr/share/edk2-ovmf/OVMF_CODE.secboot.fd",“node-name”:“libvirt-pflash0-storage”,“auto-read-only”:true,“discard”:“unmap”}’
-blockdev ‘{“node-name”:“libvirt-pflash0-format”,“read-only”:true,“driver”:“raw”,“file”:“libvirt-pflash0-storage”}’
-blockdev ‘{“driver”:“file”,“filename”:"/var/lib/libvirt/qemu/nvram/W11x64_VARS.fd",“node-name”:“libvirt-pflash1-storage”,“auto-read-only”:true,“discard”:“unmap”}’
-blockdev ‘{“node-name”:“libvirt-pflash1-format”,“read-only”:false,“driver”:“raw”,“file”:“libvirt-pflash1-storage”}’
-machine pc-q35-7.2,usb=off,vmport=off,smm=on,dump-guest-core=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format
-accel kvm
-cpu host,migratable=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff
-global driver=cfi.pflash01,property=secure,value=on
-m 32768
-object ‘{“qom-type”:“memory-backend-ram”,“id”:“pc.ram”,“size”:34359738368}’
-overcommit mem-lock=off
-smp 8,sockets=1,cores=4,threads=2
-uuid 9c83e463-97f8-4d87-b8a8-32d8b4e4c3b1
-no-user-config
-nodefaults
-chardev socket,id=charmonitor,fd=27,server=on,wait=off
-mon chardev=charmonitor,id=monitor,mode=control
-rtc base=localtime,driftfix=slew
-global kvm-pit.lost_tick_policy=delay
-no-hpet
-no-shutdown
-global ICH9-LPC.disable_s3=1
-global ICH9-LPC.disable_s4=1
-boot menu=on,strict=on
-device ‘{“driver”:“pcie-root-port”,“port”:16,“chassis”:1,“id”:“pci.1”,“bus”:“pcie.0”,“multifunction”:true,“addr”:“0x2”}’
-device ‘{“driver”:“pcie-root-port”,“port”:17,“chassis”:2,“id”:“pci.2”,“bus”:“pcie.0”,“addr”:“0x2.0x1”}’
-device ‘{“driver”:“pcie-root-port”,“port”:18,“chassis”:3,“id”:“pci.3”,“bus”:“pcie.0”,“addr”:“0x2.0x2”}’
-device ‘{“driver”:“pcie-root-port”,“port”:19,“chassis”:4,“id”:“pci.4”,“bus”:“pcie.0”,“addr”:“0x2.0x3”}’
-device ‘{“driver”:“pcie-root-port”,“port”:20,“chassis”:5,“id”:“pci.5”,“bus”:“pcie.0”,“addr”:“0x2.0x4”}’
-device ‘{“driver”:“pcie-root-port”,“port”:21,“chassis”:6,“id”:“pci.6”,“bus”:“pcie.0”,“addr”:“0x2.0x5”}’
-device ‘{“driver”:“pcie-root-port”,“port”:22,“chassis”:7,“id”:“pci.7”,“bus”:“pcie.0”,“addr”:“0x2.0x6”}’
-device ‘{“driver”:“pcie-root-port”,“port”:23,“chassis”:8,“id”:“pci.8”,“bus”:“pcie.0”,“addr”:“0x2.0x7”}’
-device ‘{“driver”:“pcie-root-port”,“port”:24,“chassis”:9,“id”:“pci.9”,“bus”:“pcie.0”,“multifunction”:true,“addr”:“0x3”}’
-device ‘{“driver”:“pcie-root-port”,“port”:25,“chassis”:10,“id”:“pci.10”,“bus”:“pcie.0”,“addr”:“0x3.0x1”}’
-device ‘{“driver”:“pcie-root-port”,“port”:26,“chassis”:11,“id”:“pci.11”,“bus”:“pcie.0”,“addr”:“0x3.0x2”}’
-device ‘{“driver”:“pcie-root-port”,“port”:27,“chassis”:12,“id”:“pci.12”,“bus”:“pcie.0”,“addr”:“0x3.0x3”}’
-device ‘{“driver”:“pcie-root-port”,“port”:28,“chassis”:13,“id”:“pci.13”,“bus”:“pcie.0”,“addr”:“0x3.0x4”}’
-device ‘{“driver”:“pcie-root-port”,“port”:29,“chassis”:14,“id”:“pci.14”,“bus”:“pcie.0”,“addr”:“0x3.0x5”}’
-device ‘{“driver”:“pcie-root-port”,“port”:30,“chassis”:15,“id”:“pci.15”,“bus”:“pcie.0”,“addr”:“0x3.0x6”}’
-device ‘{“driver”:“pcie-pci-bridge”,“id”:“pci.16”,“bus”:“pci.1”,“addr”:“0x0”}’
-device ‘{“driver”:“qemu-xhci”,“p2”:15,“p3”:15,“id”:“usb”,“bus”:“pci.3”,“addr”:“0x0”}’
-device ‘{“driver”:“lsi”,“id”:“scsi0”,“bus”:“pci.16”,“addr”:“0x1”}’
-device ‘{“driver”:“virtio-serial-pci”,“id”:“virtio-serial0”,“bus”:“pci.4”,“addr”:“0x0”}’
-blockdev ‘{“driver”:“file”,“filename”:"/vdisks/boot/VM-W11x64/VM-W11x64.img",“node-name”:“libvirt-2-storage”,“auto-read-only”:true,“discard”:“unmap”}’
-blockdev ‘{“node-name”:“libvirt-2-format”,“read-only”:false,“driver”:“raw”,“file”:“libvirt-2-storage”}’
-device ‘{“driver”:“virtio-blk-pci”,“bus”:“pci.5”,“addr”:“0x0”,“drive”:“libvirt-2-format”,“id”:“virtio-disk0”,“bootindex”:1}’
-blockdev ‘{“driver”:“file”,“filename”:"/vdisks/storage/ISO/lxd-Win11_22H2_English_x64v1.iso",“node-name”:“libvirt-1-storage”,“auto-read-only”:true,“discard”:“unmap”}’
-blockdev ‘{“node-name”:“libvirt-1-format”,“read-only”:true,“driver”:“raw”,“file”:“libvirt-1-storage”}’
-device ‘{“driver”:“ide-cd”,“bus”:“ide.0”,“drive”:“libvirt-1-format”,“id”:“sata0-0-0”}’
-netdev tap,fd=28,vhost=on,vhostfd=30,id=hostnet0
-device ‘{“driver”:“virtio-net-pci”,“netdev”:“hostnet0”,“id”:“net0”,“mac”:“52:54:00:ea:d3:03”,“bus”:“pci.2”,“addr”:“0x0”}’
-chardev pty,id=charserial0
-device ‘{“driver”:“isa-serial”,“chardev”:“charserial0”,“id”:“serial0”,“index”:0}’
-chardev spicevmc,id=charchannel0,name=vdagent
-device ‘{“driver”:“virtserialport”,“bus”:“virtio-serial0.0”,“nr”:1,“chardev”:“charchannel0”,“id”:“channel0”,“name”:“com.redhat.spice.0”}’
-chardev socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/12-W11x64-swtpm.sock
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm
-device ‘{“driver”:“tpm-crb”,“tpmdev”:“tpm-tpm0”,“id”:“tpm0”}’
-device ‘{“driver”:“usb-tablet”,“id”:“input0”,“bus”:“usb.0”,“port”:“1”}’
-audiodev ‘{“id”:“audio1”,“driver”:“spice”}’
-spice port=5900,addr=127.0.0.1,disable-ticketing=on,seamless-migration=on
-device ‘{“driver”:“qxl-vga”,“id”:“video0”,“max_outputs”:1,“ram_size”:67108864,“vram_size”:67108864,“vram64_size_mb”:0,“vgamem_mb”:16,“bus”:“pcie.0”,“addr”:“0x1”}’
-device ‘{“driver”:“ich9-intel-hda”,“id”:“sound0”,“bus”:“pcie.0”,“addr”:“0x1b”}’
-device ‘{“driver”:“hda-duplex”,“id”:“sound0-codec0”,“bus”:“sound0.0”,“cad”:0,“audiodev”:“audio1”}’
-chardev spicevmc,id=charredir0,name=usbredir
-device ‘{“driver”:“usb-redir”,“chardev”:“charredir0”,“id”:“redir0”,“bus”:“usb.0”,“port”:“2”}’
-chardev spicevmc,id=charredir1,name=usbredir
-device ‘{“driver”:“usb-redir”,“chardev”:“charredir1”,“id”:“redir1”,“bus”:“usb.0”,“port”:“3”}’
-device ‘{“driver”:“virtio-balloon-pci”,“id”:“balloon0”,“bus”:“pci.6”,“addr”:“0x0”}’
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny
-msg timestamp=on

LXD/QEMU:

Machine

[machine]
graphics = “off”
type = “q35”
accel = “kvm”
usb = “off”

[global]
driver = “ICH9-LPC”
property = “disable_s3”
value = “1”

[global]
driver = “ICH9-LPC”
property = “disable_s4”
value = “1”

[boot-opts]
strict = “on”

Memory

[memory]
size = “32768M”

CPU

[smp-opts]
cpus = “1”
maxcpus = “48”

[object “mem0”]
qom-type = “memory-backend-memfd”
size = “32768M”
share = “on”

[numa]
type = “node”
nodeid = “0”
memdev = “mem0”

Firmware (read only)

[drive]
file = “/usr/share/edk2-ovmf/OVMF_CODE.fd”
if = “pflash”
format = “raw”
unit = “0”
readonly = “on”

Firmware settings (writable)

[drive]
file = “/dev/fd/3”
if = “pflash”
format = “raw”
unit = “1”

Qemu control

[chardev “monitor”]
backend = “socket”
path = “/var/log/lxd/VM-Win11x64/qemu.monitor”
server = “on”
wait = “off”

[mon]
chardev = “monitor”
mode = “control”

Console

[chardev “console”]
backend = “socket”
path = “/var/log/lxd/VM-Win11x64/qemu.console”
server = “on”
wait = “off”

[device “qemu_pcie0”]
driver = “pcie-root-port”
bus = “pcie.0”
addr = “1.0”
chassis = “0”
multifunction = “on”

Balloon driver

[device “qemu_balloon”]
driver = “virtio-balloon-pci”
bus = “qemu_pcie0”
addr = “00.0”
multifunction = “on”

Random number generator

[object “qemu_rng”]
qom-type = “rng-random”
filename = “/dev/urandom”

[device “dev-qemu_rng”]
driver = “virtio-rng-pci”
bus = “qemu_pcie0”
addr = “00.1”
rng = “qemu_rng”

Input

[device “qemu_keyboard”]
driver = “virtio-keyboard-pci”
bus = “qemu_pcie0”
addr = “00.2”

Input

[device “qemu_tablet”]
driver = “virtio-tablet-pci”
bus = “qemu_pcie0”
addr = “00.3”

Vsock

[device “qemu_vsock”]
driver = “vhost-vsock-pci”
bus = “qemu_pcie0”
addr = “00.4”
guest-cid = “98”

Virtual serial bus

[device “dev-qemu_serial”]
driver = “virtio-serial-pci”
bus = “qemu_pcie0”
addr = “00.5”

LXD serial identifier

[chardev “qemu_serial-chardev”]
backend = “ringbuf”
size = “16B”

[device “qemu_serial”]
driver = “virtserialport”
name = “org.linuxcontainers.lxd”
chardev = “qemu_serial-chardev”
bus = “dev-qemu_serial.0”

Spice agent

[chardev “qemu_spice-chardev”]
backend = “spicevmc”
name = “vdagent”

[device “qemu_spice”]
driver = “virtserialport”
name = “com.redhat.spice.0”
chardev = “qemu_spice-chardev”
bus = “dev-qemu_serial.0”

Spice folder

[chardev “qemu_spicedir-chardev”]
backend = “spiceport”
name = “org.spice-space.webdav.0”

[device “qemu_spicedir”]
driver = “virtserialport”
name = “org.spice-space.webdav.0”
chardev = “qemu_spicedir-chardev”
bus = “dev-qemu_serial.0”

USB controller

[device “qemu_usb”]
driver = “qemu-xhci”
bus = “qemu_pcie0”
addr = “00.6”
p2 = “8”
p3 = “8”

[chardev “qemu_spice-usb-chardev1”]
backend = “spicevmc”
name = “usbredir”

[device “qemu_spice-usb1”]
driver = “usb-redir”
chardev = “qemu_spice-usb-chardev1”

[chardev “qemu_spice-usb-chardev2”]
backend = “spicevmc”
name = “usbredir”

[device “qemu_spice-usb2”]
driver = “usb-redir”
chardev = “qemu_spice-usb-chardev2”

[chardev “qemu_spice-usb-chardev3”]
backend = “spicevmc”
name = “usbredir”

[device “qemu_spice-usb3”]
driver = “usb-redir”
chardev = “qemu_spice-usb-chardev3”

[device “qemu_pcie1”]
driver = “pcie-root-port”
bus = “pcie.0”
addr = “1.1”
chassis = “1”

SCSI controller

[device “qemu_scsi”]
driver = “virtio-scsi-pci”
bus = “qemu_pcie1”
addr = “00.0”

[device “qemu_pcie2”]
driver = “pcie-root-port”
bus = “pcie.0”
addr = “1.2”
chassis = “2”

Config drive (9p)

[fsdev “qemu_config”]
fsdriver = “local”
security_model = “none”
readonly = “on”
path = “/var/lib/lxd/devices/VM-Win11x64/config.mount”

[device “dev-qemu_config-drive-9p”]
driver = “virtio-9p-pci”
bus = “qemu_pcie2”
addr = “00.0”
multifunction = “on”
mount_tag = “config”
fsdev = “qemu_config”

Config drive (virtio-fs)

[chardev “qemu_config”]
backend = “socket”
path = “/var/log/lxd/VM-Win11x64/virtio-fs.config.sock”

[device “dev-qemu_config-drive-virtio-fs”]
driver = “vhost-user-fs-pci”
bus = “qemu_pcie2”
addr = “00.1”
tag = “config”
chardev = “qemu_config”

[device “qemu_pcie3”]
driver = “pcie-root-port”
bus = “pcie.0”
addr = “1.3”
chassis = “3”

GPU

[device “qemu_gpu”]
driver = “virtio-vga”
bus = “qemu_pcie3”
addr = “00.0”

[device “qemu_pcie4”]
driver = “pcie-root-port”
bus = “pcie.0”
addr = “1.4”
chassis = “4”

[chardev “qemu_tpm-chardev_vtpm”]
backend = “socket”
path = “/var/lib/lxd/virtual-machines/VM-Win11x64/tpm.vtpm/swtpm-vtpm.sock”

[tpmdev “qemu_tpm-tpmdev_vtpm”]
type = “emulator”
chardev = “qemu_tpm-chardev_vtpm”

[device “dev-lxd_vtpm”]
driver = “tpm-crb”
tpmdev = “qemu_tpm-tpmdev_vtpm”

[device “qemu_pcie5”]
driver = “pcie-root-port”
bus = “pcie.0”
addr = “1.5”
chassis = “5”

[device “qemu_pcie6”]
driver = “pcie-root-port”
bus = “pcie.0”
addr = “1.6”
chassis = “6”

[device “qemu_pcie7”]
driver = “pcie-root-port”
bus = “pcie.0”
addr = “1.7”
chassis = “7”

[device “qemu_pcie8”]
driver = “pcie-root-port”
bus = “pcie.0”
addr = “2.0”
chassis = “8”
multifunction = “on”

Since there doesn’t seem to be any troubleshooting guidance for this lxd/qemu issue and the issue does not exist when leveraging libvirt/qemu, the most appropriate solution seems: use libvirt for Win11 VMs.

Hi, I’ve just been battling with WS2022 ISO and LXD Windows Server 2022 images on a repurposed Dell R740xd with Intel 6148… both ISO and images froze during boot.

Using CPU Passthrough in raw.qemu worked around this issue. Maybe it’ll help you.

Also had an issue where WS2022 VMs would crash regularly and the following modprobe workaround worked for me on older R620 servers which did actually support tdp_mmu:
https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0#Older_Hardware_and_New_5.15_Kernel

Actually, not related booting problem but if your windows vm closed in random times, you should set kvm.tdp_mmu=N in your host as @mratt stated.
Regards.

@cemzafer Agree… the raw.qemu: -cpu host in the instance’s config, got WS2022 ISO and instances from an image working on this CPU, opposed to them freezing a few seconds into the boot.

The kvm.tdp_mmu modprobe config on the host, has been required on this CPU and a number of older ones as well, which do support tdp_mmu / Intel PET, but end up randomly crashing with hardware error 0x80000021. I would be much happier of this could be done at the instance level, or even fixed.

I’ve been able to boot and install from a WS2019 ISO fine without the above settings but not really tested it as a guest OS, because I’m focussing on a complete migration… new conventions, domain, WS2022, etc due to mergers.

setting the CPU passthrough works for me, this is the command I used in case someone is not familiar with the actual command:

lxc config set win11 raw.qemu="-cpu host"

later I was able to boot win11 without setting CPU passthrough or kvm.tdp_mmu, I just have to press Enter when prompted press any key to boot from CD-ROM...