LXD Refuses To Boot Valid Windows 11 QEMU VM

LXD
,
1

Greetings all! I’ve been fighting with LXD for about a week now trying to work this out on my own, but I think I’ve finally exhausted the advice I’ve been able to find on here and on the internet in general without asking a more specific question.

The short version is that I’m trying to get a Windows 11 (arm64) VM spun up. My hardware is a three-node cluster built from 8GB Raspberry Pi 4 units running Ubuntu 21.10 64-bit server (arm64). LXD/LXC run just fine and will spin up containers and linux-based VM images without complaint or issue.

After much head-against-proverbial-wall banging, I was able to get a fully functional VM spun up in QEMU on one of the cluster nodes like this:

sudo qemu-system-aarch64 \
    -cpu host \
    -enable-kvm \
    -M virt \
    -m 4096 \
    -mem-prealloc \
    -parallel none \
    -serial stdio \
    -device ramfb \
    -device qemu-xhci -device usb-tablet -device usb-kbd \
    -name "Win11 (arm64)" \
    -smp sockets=1,cores=4,threads=1 \
    -bios /images/QEMU_EFI.img \
    -rtc base=localtime,clock=host,driftfix=none \
    -device nvme,drive=lxd_root,serial=lxd-root \
    -drive file=/images/win11.qcow2,if="none",id=lxd_root \
    -drive file=/images/win11-pro_en-us_arm64.iso,if=none,id=install,media=cdrom \
    -device usb-storage,drive=install,removable=true,bus=usb-bus.0 \
    -drive file=/images/virtio-win-0.1.208.iso,if=none,id=drivers,media=cdrom \
    -device usb-storage,drive=drivers,removable=true,bus=usb-bus.0 \
    -drive file=/images/spice-guest-tools-0.164.2.iso,if=none,id=spice-tools,media=cdrom \
    -device usb-storage,drive=spice-tools,removable=true,bus=usb-bus.0 \
    -net nic,model=virtio \
    -net user

Once the install was complete and the drivers installed, you can (obviously) omit the extra -drive and -device arguments for the driver disks and the VM boots and runs as expected, fully functional to all of my testing so far.

My thought (initially) was that it I’d be able to get the VM “initialized” and then just drop in into LXD so to speak. That absolutely does not work, for whatever reason. I’ve tried it several different ways, but for brevity the two most relevant are:

architecture: aarch64
config:
  boot.autostart: "false"
  limits.cpu: "4"
  limits.memory: 4GB
  security.secureboot: "false"
  volatile.eth0.hwaddr: 00:16:3e:68:a9:75
  volatile.last_state.power: STOPPED
  volatile.uuid: 024e134b-2fcc-4ae2-af58-312ff43e929c
  volatile.vsock_id: "21"
devices:
  win-root:
    source: /images/win11.qcow2
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: "Windows 11 (arm64)"

and

architecture: aarch64
config:
  boot.autostart: "false"
  limits.cpu: "4"
  limits.memory: 4GB
  security.secureboot: "false"
  volatile.eth0.hwaddr: 00:16:3e:68:a9:75
  volatile.last_state.power: STOPPED
  volatile.uuid: 024e134b-2fcc-4ae2-af58-312ff43e929c
  volatile.vsock_id: "21"
  raw.qemu: -drive file=/var/lib/snapd/hostfs/images/win11.qcow2,if=none,id=win_root -device nvme,drive=win_root,serial=win-root
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: "Windows 11 (arm64)"

Neither one will actually boot, though they don’t appear to exhibit different behavior - they either throw an error about not being able to boot from some LXC-created device (not the QCOW2 image) or (if I manually enter the EFI shell and boot from the correct entry) just sit at a black screen and do nothing.

Notes:

  • In order to get the second configuration working at all, I had to completely disable AppArmor and re-install the LXD snap in --devmode which seems less than ideal. This was due to the fact that no matter what I tried, LXD/AppArmor refused to allow QEMU access to literally anything almost no matter where I put it path-wise
  • The EFI firmware package available via apt may or may not be out of date, I can’t really tell. I just know I didn’t have any real luck getting anything to boot (Windows-wise) without using a more recent EDK2 build from here.
    • Getting ^ that working involved “replacing” the files referenced by LXD by issuing a custom bind mount: mount --bind -o nodev,ro /images/qemu /snap/lxd/22151/share/qemu/
  • The “standard” configuration looking something like:
    devices:
    drivers:
        source: /images/virtio-win-0.1.208.iso
        type: disk
    install:
        boot.priority: "10"
        source: /images/win11-pro_en-us_arm64.iso
        type: disk
    root:
        path: /
        pool: local
        size: 64GB
        type: disk
    spice:
        source: /images/spice-guest-tools-0.164.2.iso
        type: disk
    gets me to the same point as the other two above - a black screen with no activity. There’s absolutely no change in that behavior even if I use distrobuilder to pre-patch the drivers into the install image.

tl;dr I can’t shake the feeling that I’ve just missed something incredibly simple in a forest-through-the-trees sort of deal, and I’m hoping that someone here who knows more about it will be able to point me in the right direction.