@stgraber Thanks for the suggestion. It helped me out a lot despite being a bit finicky. Long story short, I have succeeded in running a macOS VM inside LXD.
I first started out by getting your suggestion to work which required me to:
nsenter --mount=/run/snapd/ns/lxd.mntns/lxd.mnt cp /snap/lxd/current/bin/qemu-system-x86_64 /tmp/qemu-system-x86_64
mount -o bind /home/anderson/qemu-system-x86_64.sh /snap/lxd/current/bin/qemu-system-x86_64
lxc config set macOS raw.apparmor "/tmp/** mrixkw, /usr/bin/** rixk, /usr/lib/** rixk,"
At this point I am able to dump and hack around the qemu-system-x86_64
command that is executed by LXD. I didn’t end up making much progress with this until I realized this morning that I could copy the qemu-system-x86_64
binary out of the snap and into the Ubuntu Focal environment where I originally setup the VM. After doing so I determined that the QEMU 6.2.0 that is packaged with LXD is absolutely capable (i.e. no missing compile time features, etc).
So, now that I am confident the packaged QEMU can work I went back to tinkering with the QEMU CLI flags as well as the LXD generated qemu.conf
and have found the following changes are required:
--- qemu.conf.orig 2022-01-05 13:40:46.843264685 -0800
+++ qemu.conf.new 2022-01-05 14:56:55.619618592 -0800
@@ -4,7 +4,7 @@
graphics = "off"
type = "q35"
accel = "kvm"
-usb = "off"
+usb = "on"
[global]
driver = "ICH9-LPC"
@@ -110,6 +110,8 @@
addr = "00.2"
+[device "usb_keyboard"]
+driver = "usb-kbd"
# Input
[device "qemu_tablet"]
@@ -118,6 +120,8 @@
addr = "00.3"
+[device "usb_tablet"]
+driver = "usb-tablet"
# Vsock
[device "qemu_vsock"]
@@ -213,6 +217,10 @@
bus = "qemu_pcie1"
addr = "00.0"
+# SATA controller
+[device "sata"]
+driver = "ich9-ahci"
+
[device "qemu_pcie2"]
@@ -258,12 +266,8 @@
# GPU
-[device "qemu_gpu"]
-driver = "virtio-vga"
-bus = "qemu_pcie3"
-addr = "00.0"
-
-
+[device "qemu_vga"]
+driver = "VGA"
[device "qemu_pcie4"]
driver = "pcie-root-port"
@@ -285,11 +289,8 @@
readonly = "off"
[device "dev-lxd_open-core-boot"]
-driver = "scsi-hd"
-bus = "qemu_scsi.0"
-channel = "0"
-scsi-id = "0"
-lun = "1"
+driver = "ide-hd"
+bus = "sata.2"
drive = "lxd_open-core-boot"
bootindex = "0"
@@ -307,11 +308,8 @@
readonly = "off"
[device "dev-lxd_root"]
-driver = "scsi-hd"
-bus = "qemu_scsi.0"
-channel = "0"
-scsi-id = "1"
-lun = "1"
+driver = "ide-hd"
+bus = "sata.4"
drive = "lxd_root"
bootindex = "1"
@@ -337,9 +335,3 @@
chassis = "7"
-[device "qemu_pcie8"]
-driver = "pcie-root-port"
-bus = "pcie.0"
-addr = "2.0"
-chassis = "8"
-multifunction = "on"
The changes boil down to 3 primary differences:
- Add an
ich9-ahci
SATA controller and connect the block devices to it instead of SCSI. Remove qemu_pcie8
since it conflicts with the SATA controller.
- Switch the GPU from PCIe
virtio-vga
to PCI VGA
- Enable USB and add
usb-kbd
and usb-tablet
devices so keyboard and mouse work over SPICE.
In addition to the qemu.conf
changes the following extra settings are needed in the LXD config for the VM:
config:
raw.qemu: -cpu host,vendor=GenuineIntel -device isa-applesmc,osk=<redacted>
security.secureboot: "false"
devices:
open-core-boot:
boot.priority: "2"
pool: zfs
source: open-core-boot
type: disk
- The
-cpu host,vendor=GenuineIntel
override is to prevent OpenCore from hanging on my Ryzen9. It likely isn’t necessary for hosts that have Intel CPUs.
- The
-device isa-applesmc
override is required for macOS to boot. It is discussed plenty around the Internet and should be left as an override for obvious reasons.
- Since the boot process requires two block devices I put OpenCore into a custom volume and forced a high priority to give it
bootindex = "0"
. This change is nice because it prevents really long boot delays due to PXE boot options normally coming before custom volumes.
So, what I am hoping for is maybe a few extra LXD configuration options to address the 3 variations covered in the diff above. For example:
config:
qemu.gpu = "pci-vga"
qemu.block.driver = "sata"
qemu.inputs = "usb"
I don’t think there is anyway I can accomplish all of these via raw.qemu
because it goes after the -readconfig
flag and the entire scheme doesn’t seem to support a notion of later flags being able to delete and/or override previous options.
Any chance we can see these 3 tweaks via configuration options so users can host macOS VMs using LXD?