FreeBSD 13.1 Kernel panic in VM


My OPNSense 22.1 (FreeBSD 13.0) VM works great, but runs into a kernel panic after upgrade to OPNSense 22.7 (FreeBSD 13.1). Installer ISO also runs into the same kernel panic.

I want to change the machine type to i440fx to debug but I can’t find a valid list of supported qemu machine types anywhere. Who can help me out? :slight_smile:

Running LXD 5.5, secureboot set to off for this VM. I’ve tried an older Q35 machine mode with -machine pc-q35-2.6 but that doesnt work either.

acpi_syscontainer2: <System Container> port 0x620-0x62f on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 14.0.
psm0: model IntelliMouse Explorer, device ID 4
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
attimer0: <AT timer> at port 0x40 on isa0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
kernel trap 18 with interrupts disabled

Fatal trap 18: integer divide fault while in kernel mode
cpuid = 1; apic id = 01
instruction pointer     = 0x20:0xffffffff81226bfd
stack pointer           = 0x28:0xffffffff824c1f20
frame pointer           = 0x28:0xffffffff824c1f20
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 0 (swapper)
trap number             = 18
panic: integer divide fault
cpuid = 1
time = 1
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffffff824c1d40
vpanic() at vpanic+0x17f/frame 0xffffffff824c1d90
panic() at panic+0x43/frame 0xffffffff824c1df0
trap_fatal() at trap_fatal+0x385/frame 0xffffffff824c1e50
calltrap() at calltrap+0x8/frame 0xffffffff824c1e50
--- trap 0x12, rip = 0xffffffff81226bfd, rsp = 0xffffffff824c1f20, rbp = 0xffffffff824c1f20 ---
lapic_et_start() at lapic_et_start+0x2ad/frame 0xffffffff824c1f20
configtimer() at configtimer+0x320/frame 0xffffffff824c1f70
cpu_initclocks_bsp() at cpu_initclocks_bsp+0x7f6/frame 0xffffffff824c1fa0
cpu_initclocks() at cpu_initclocks+0x20/frame 0xffffffff824c1fc0
initclocks() at initclocks+0x20/frame 0xffffffff824c1fd0
mi_startup() at mi_startup+0xdf/frame 0xffffffff824c1ff0
btext() at btext+0x22
KDB: enter: panic
[ thread pid 0 tid 100000 ]
Stopped at      kdb_enter+0x37: movq    $0,0x11f81be(%rip)

Yeah, I’ve seen the same here, that’s why we’ve not published any guides on running FreeBSD or OpenBSD based OS in LXD at this point.

I don’t know exactly what the problem is either… You’d think that with Q35 being roughly a Core 2 Duo era machine, that BSD could handle it.

Do you have a list of machine types I could try? I was not able to run the qemu binary from the snap directory and could not chroot into the snap either :slight_smile:

I suspect the list of options is likely:

microvm              microvm (i386)
pc                   Standard PC (i440FX + PIIX, 1996) (alias of pc-i440fx-6.2)
pc-i440fx-6.2        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-6.1        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-6.0        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-5.2        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-5.1        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-5.0        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-4.2        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-4.1        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-4.0        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-3.1        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-3.0        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.9        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.8        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.7        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.6        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.5        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.4        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.3        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.2        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.12       Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.11       Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.10       Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.1        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.0        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.7        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.6        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.5        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.4        Standard PC (i440FX + PIIX, 1996)
q35                  Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-6.2)
pc-q35-6.2           Standard PC (Q35 + ICH9, 2009)
pc-q35-6.1           Standard PC (Q35 + ICH9, 2009)
pc-q35-6.0           Standard PC (Q35 + ICH9, 2009)
pc-q35-5.2           Standard PC (Q35 + ICH9, 2009)
pc-q35-5.1           Standard PC (Q35 + ICH9, 2009)
pc-q35-5.0           Standard PC (Q35 + ICH9, 2009)
pc-q35-4.2           Standard PC (Q35 + ICH9, 2009)
pc-q35-4.1           Standard PC (Q35 + ICH9, 2009)
pc-q35-4.0.1         Standard PC (Q35 + ICH9, 2009)
pc-q35-4.0           Standard PC (Q35 + ICH9, 2009)
pc-q35-3.1           Standard PC (Q35 + ICH9, 2009)
pc-q35-3.0           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.9           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.12          Standard PC (Q35 + ICH9, 2009)
pc-q35-2.11          Standard PC (Q35 + ICH9, 2009)
pc-q35-2.10          Standard PC (Q35 + ICH9, 2009)
isapc                ISA-only PC
none                 empty machine
x-remote             Experimental remote machine

A while back I had this solved in the #lxd channel.

The issue with freebsd 13.1 is with some escoteric feature that was altered in somewhere early lxd5 (pv_passthrough?) .

Try shutting down the VM and adding migration.stateful: “true” to the instances config.

Nice to see others running the BSDs. Someday the lxd VM agent will wind up there hint hint

1 Like

Getting the agent should be pretty straightforward.
You’ll need to build it by hand rather than running the binary available in the 9p share, but there shouldn’t be much Linux-specific logic in there.

1 Like

FWIW I am not super happy with this as a solution since it’s buried in code that is moving.
I will try to find the #lxd thread with the specifics that led to this solution in case this changes again.

The real fix should be to have the BSD kernel not crash :wink:

#lxc 11:18 < feurig> So am trying an experiment at work. Last year I ran up a freebsd 12.2 box so I could interview for my current job. I upgraded the vm to 13.0 with no problems and
then I upgraded the underlying lxd from 4 to 5.2 also with no problems.
11:20 < feurig> When I tried a fresh install with the currently supported freebsd it cores. When I upgraded my working vm to 13.1 it also cores.
11:21 < feurig> So the kernel in 13.1 isnt compatible with the current vm. (though we are running 13.1 on qemu kvms outside of lxd). And freebsd prolly doesnt care because they are
providing virtual machine images.
11:22 < feurig> My quandry is which would be more painful.
11:22 < feurig> 1. Working with the freebsd community to figure out what they broke or how to install with the old kernel.
11:22 < feurig> 2. Working with the lxd community to figure out how to unbundle the vmdk that freebsd is providing…
11:22 < feurig> any thoughts?

This is a working vm config. (boot.priority altered post install).

I am more concerned at subtle changes in lxd effecting running systems than I am with fixing the BSD kernel.

lxc config show kif
architecture: x86_64
  migration.stateful: "true"
  raw.apparmor: /install/** rwk,
  raw.qemu: -boot menu=on -machine pc-q35-2.6
  security.secureboot: "false" cace7bcf-023f-4a52-81eb-cedfc5ffe21f
  volatile.eth0.host_name: tapb7987e3e
  volatile.eth0.hwaddr: xx:xx:xx:xx:xx:xx
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: d3d7763d-0fcc-41a4-897a-dfc9ca7d4535
  volatile.vsock_id: "11"
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
    boot.priority: "0"
    source: /install/FreeBSD-13.1-RELEASE-amd64-dvd1.iso
    type: disk
    boot.priority: "10"
    path: /
    pool: local
    size: 80GB
    size.state: 16GB
    type: disk
ephemeral: false
- default
stateful: false
description: "Working VM configuration for Freebsde on LXD"

Oh, thank God! I was just trying to get OPNsense 22.7 and latest snapshot of pfSense running and I couldn’t make anything work until you mentioned having to add migration.stateful=true

I actually found that just enabling stateful migration was enough; the raw.qemu-settings or raw.apparmor-stuff weren’t needed, the VMs boot up just fine without.

1 Like

We have a video on running FreeBSD now btw:

1 Like

I think you were referring to this video:

Yep its the same one I linked to

Oh, for some reason I can’t see your link :confused:

This post seems like it could be useful to people trying to use BSD VMs too:

1 Like

I was thinking I’d just combine this information into a brief tutorial in the tutorials-section soonish. I’m testing a few things first.

1 Like

Here’s the tutorial