Premise:
Currently, starting with FreeBSD 13.1, the kernel throws a panic at boot with the default settings under LXD. This also prevents distributions using recent FreeBSD kernels from booting as well, including OPNsense 22.7 and pfSense 2.7.0.
The issue appears to stem from the fact that by default, LXD tells QEMU to passthrough all supported Hyper-V enlightenments to the virtual-machines and that, presumably, causes FreeBSD’s kernel to interpret the CPU wrong.
Solution:
You can prevent LXD from passing through Hyper-V enlightenments by overriding the relevant part of the QEMU command used to launch the virtual machine by setting raw.qemu to -c cpu host, ie. the basic flow is as follows:
# Initialize empty VM. Give it CPU-cores, memory and set root disk size:
# Secure boot needs to be disabled too.
lxc init freebsd131 \
--vm \
--empty \
-c limits.cpu=2 \
-c limits.memory=2GiB \
-c security.secureboot=false \
-d root,size=8GiB
# Set the qemu-override allowing the FreeBSD-system to boot successfully:
echo '-cpu host' | lxc config set freebsd131 raw.qemu -
# Add the device to install from to the virtual machine:
lxc config device add freebsd131 iso disk \
source=/FreeBSD-13.1-RELEASE-amd64-memstick.img \
boot.priority=10
You’ll need virt-viewer installed to be able to see the installer. After you have that installed, you can proceed as follows:
lxc start freebsd131
lxc console freebsd131 --type=vga &
Install as usual and when you reboot the system, it’ll boot back to the installer. You’ll need to stop the VM and then remove the installer disk from the VM:
lxc stop -f freebsd131
lxc config device remove freebsd131 iso
Currently remaining issues:
FreeBSD hogs one CPU-core at 100% as of writing this. The issue stems from LXD by default adding a virtio-rng device to the VM and FreeBSD doing something wrong with it. This can be sidestepped inside the guest by adding the following line to /etc/rc.conf (/etc/rc.conf.local for pfSense):
devmatch_blacklist="virtio_random.ko"
Another option is to prevent LXD from passing the virtio-rng device to the virtual machine in the first place, thereby not needing any guest-side setup:
echo '[device "dev-qemu_rng"]' | lxc config set freebsd131 raw.qemu.conf -
This workaround can hopefully be disabled at some point with the FreeBSD-developers patching whatever is the underlying cause.
Example configuration:
With the above workarounds applied, your configuration would probably look something akin to the following:
architecture: x86_64
config:
limits.cpu: 0,1
limits.memory: 2GiB
raw.qemu: |
-cpu host
raw.qemu.conf: |
[device "dev-qemu_rng"]
security.secureboot: "false"
volatile.cloud-init.instance-id: 406f4fcd-7dbf-4b11-9aa0-27e5bd735c1a
volatile.eth0.hwaddr: 00:16:3e:df:3f:9e
volatile.last_state.power: STOPPED
volatile.last_state.ready: "false"
volatile.uuid: e9ab30f1-9bba-4e10-8b27-63a136a3d638
volatile.vsock_id: "114"
devices:
root:
path: /
pool: default
size: 10GiB
type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""
Other:
Credits to @delmar for leading me to figuring out how to get FreeBSD 13.1 to boot. Delmar’s solution was almost perfect, but it prevented me from passing through physical devices to the VM, whereas overriding qemu command like here does not. @sdeziel also pointed out to me that you can actually prevent LXD from adding the virtio-rng device in the first place, instead of necessarily having to use guest-side tweaks.