macOS: Can I test custom qemu-system-x86_64 command inside LXD snap?

I have been experimenting with getting LXD to run macOS inside a virtual machine. I have made some progress, but it has been slow with many issues that are hard to hack around using the raw.qemu option and manually copying files into the LXD mount namespace so they are even usable.

I am guessing the primary complication here is that LXD only uses SCSI for the disks and the projects that have macOS running on QEMU 4+ are using SATA because that is the only interface type with a working kext driver inside the VM. I have managed to get the VM to successfully boot into OpenCore which can see and start the macOS disk, however it hangs early in the kernel messages, likely due to the boot disk being SCSI instead of SATA (i.e. no kext support for OS on SCSI).

I would like to test a known working qemu-system-x86_64 command line inside of the LXD snad to see whether or not the QEMU 6.2 packaged with LXD can successfully run the VM. Is there a way to “get inside the snap” and manually try a QEMU command line?

You can bind-mount stuff on top of what’s in the snap, though that can get a bit tricky when libraries get involved.

Basically you can do mount -o bind /your/own/file /snap/lxd/current/bin/something and that should propagate to the snap and have it run your version of the binary.

@stgraber Thanks for the suggestion. It helped me out a lot despite being a bit finicky. Long story short, I have succeeded in running a macOS VM inside LXD.

I first started out by getting your suggestion to work which required me to:

nsenter --mount=/run/snapd/ns/lxd.mntns/lxd.mnt cp /snap/lxd/current/bin/qemu-system-x86_64 /tmp/qemu-system-x86_64
mount -o bind /home/anderson/qemu-system-x86_64.sh /snap/lxd/current/bin/qemu-system-x86_64
lxc config set macOS raw.apparmor "/tmp/** mrixkw, /usr/bin/** rixk, /usr/lib/** rixk,"

At this point I am able to dump and hack around the qemu-system-x86_64 command that is executed by LXD. I didn’t end up making much progress with this until I realized this morning that I could copy the qemu-system-x86_64 binary out of the snap and into the Ubuntu Focal environment where I originally setup the VM. After doing so I determined that the QEMU 6.2.0 that is packaged with LXD is absolutely capable (i.e. no missing compile time features, etc).

So, now that I am confident the packaged QEMU can work I went back to tinkering with the QEMU CLI flags as well as the LXD generated qemu.conf and have found the following changes are required:

--- qemu.conf.orig	2022-01-05 13:40:46.843264685 -0800
+++ qemu.conf.new	2022-01-05 14:56:55.619618592 -0800
@@ -4,7 +4,7 @@
 graphics = "off"
 type = "q35"
 accel = "kvm"
-usb = "off"
+usb = "on"
 
 [global]
 driver = "ICH9-LPC"
@@ -110,6 +110,8 @@
 addr = "00.2"
 
 
+[device "usb_keyboard"]
+driver = "usb-kbd"
 
 # Input
 [device "qemu_tablet"]
@@ -118,6 +120,8 @@
 addr = "00.3"
 
 
+[device "usb_tablet"]
+driver = "usb-tablet"
 
 # Vsock
 [device "qemu_vsock"]
@@ -213,6 +217,10 @@
 bus = "qemu_pcie1"
 addr = "00.0"
 
+# SATA controller
+[device "sata"]
+driver = "ich9-ahci"
+
 
 
 [device "qemu_pcie2"]
@@ -258,12 +266,8 @@
 
 
 # GPU
-[device "qemu_gpu"]
-driver = "virtio-vga"
-bus = "qemu_pcie3"
-addr = "00.0"
-
-
+[device "qemu_vga"]
+driver = "VGA"
 
 [device "qemu_pcie4"]
 driver = "pcie-root-port"
@@ -285,11 +289,8 @@
 readonly = "off"
 
 [device "dev-lxd_open-core-boot"]
-driver = "scsi-hd"
-bus = "qemu_scsi.0"
-channel = "0"
-scsi-id = "0"
-lun = "1"
+driver = "ide-hd"
+bus = "sata.2"
 drive = "lxd_open-core-boot"
 bootindex = "0"
 
@@ -307,11 +308,8 @@
 readonly = "off"
 
 [device "dev-lxd_root"]
-driver = "scsi-hd"
-bus = "qemu_scsi.0"
-channel = "0"
-scsi-id = "1"
-lun = "1"
+driver = "ide-hd"
+bus = "sata.4"
 drive = "lxd_root"
 bootindex = "1"
 
@@ -337,9 +335,3 @@
 chassis = "7"
 
 
-[device "qemu_pcie8"]
-driver = "pcie-root-port"
-bus = "pcie.0"
-addr = "2.0"
-chassis = "8"
-multifunction = "on"

The changes boil down to 3 primary differences:

  1. Add an ich9-ahci SATA controller and connect the block devices to it instead of SCSI. Remove qemu_pcie8 since it conflicts with the SATA controller.
  2. Switch the GPU from PCIe virtio-vga to PCI VGA
  3. Enable USB and add usb-kbd and usb-tablet devices so keyboard and mouse work over SPICE.

In addition to the qemu.conf changes the following extra settings are needed in the LXD config for the VM:

config:
  raw.qemu: -cpu host,vendor=GenuineIntel -device isa-applesmc,osk=<redacted>
  security.secureboot: "false"
devices:
  open-core-boot:
    boot.priority: "2"
    pool: zfs
    source: open-core-boot
    type: disk
  1. The -cpu host,vendor=GenuineIntel override is to prevent OpenCore from hanging on my Ryzen9. It likely isn’t necessary for hosts that have Intel CPUs.
  2. The -device isa-applesmc override is required for macOS to boot. It is discussed plenty around the Internet and should be left as an override for obvious reasons.
  3. Since the boot process requires two block devices I put OpenCore into a custom volume and forced a high priority to give it bootindex = "0". This change is nice because it prevents really long boot delays due to PXE boot options normally coming before custom volumes.

So, what I am hoping for is maybe a few extra LXD configuration options to address the 3 variations covered in the diff above. For example:

config:
  qemu.gpu = "pci-vga"
  qemu.block.driver = "sata"
  qemu.inputs = "usb"

I don’t think there is anyway I can accomplish all of these via raw.qemu because it goes after the -readconfig flag and the entire scheme doesn’t seem to support a notion of later flags being able to delete and/or override previous options.

Any chance we can see these 3 tweaks via configuration options so users can host macOS VMs using LXD?

As it still is unclear whether a feature raw.qemu.config will be introduced, right now in case of a change, the whole config object raw.qemu need to be replaced.

In classic qemu/kvm new values can be passed anytime by CLI or xml edit, some can take effect immediately, some scheduled to take effect after a restart.

Lxd rejects a raw.qemu patch whilst VM is running. How can I pass new raw.qem value to take effect after a restart?