FreeBSD 13.1 Kernel panic in VM

[vosdev (Daniel)]

Helloooo,

My OPNSense 22.1 (FreeBSD 13.0) VM works great, but runs into a kernel panic after upgrade to OPNSense 22.7 (FreeBSD 13.1). Installer ISO also runs into the same kernel panic.

I want to change the machine type to i440fx to debug but I can’t find a valid list of supported qemu machine types anywhere. Who can help me out?

Running LXD 5.5, secureboot set to off for this VM. I’ve tried an older Q35 machine mode with -machine pc-q35-2.6 but that doesnt work either.

acpi_syscontainer2: <System Container> port 0x620-0x62f on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 14.0.
psm0: model IntelliMouse Explorer, device ID 4
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
attimer0: <AT timer> at port 0x40 on isa0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
kernel trap 18 with interrupts disabled


Fatal trap 18: integer divide fault while in kernel mode
cpuid = 1; apic id = 01
instruction pointer     = 0x20:0xffffffff81226bfd
stack pointer           = 0x28:0xffffffff824c1f20
frame pointer           = 0x28:0xffffffff824c1f20
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 0 (swapper)
trap number             = 18
panic: integer divide fault
cpuid = 1
time = 1
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffffff824c1d40
vpanic() at vpanic+0x17f/frame 0xffffffff824c1d90
panic() at panic+0x43/frame 0xffffffff824c1df0
trap_fatal() at trap_fatal+0x385/frame 0xffffffff824c1e50
calltrap() at calltrap+0x8/frame 0xffffffff824c1e50
--- trap 0x12, rip = 0xffffffff81226bfd, rsp = 0xffffffff824c1f20, rbp = 0xffffffff824c1f20 ---
lapic_et_start() at lapic_et_start+0x2ad/frame 0xffffffff824c1f20
configtimer() at configtimer+0x320/frame 0xffffffff824c1f70
cpu_initclocks_bsp() at cpu_initclocks_bsp+0x7f6/frame 0xffffffff824c1fa0
cpu_initclocks() at cpu_initclocks+0x20/frame 0xffffffff824c1fc0
initclocks() at initclocks+0x20/frame 0xffffffff824c1fd0
mi_startup() at mi_startup+0xdf/frame 0xffffffff824c1ff0
btext() at btext+0x22
KDB: enter: panic
[ thread pid 0 tid 100000 ]
Stopped at      kdb_enter+0x37: movq    $0,0x11f81be(%rip)
db> 

[stgraber (Stéphane Graber)]

Yeah, I’ve seen the same here, that’s why we’ve not published any guides on running FreeBSD or OpenBSD based OS in LXD at this point.

I don’t know exactly what the problem is either… You’d think that with Q35 being roughly a Core 2 Duo era machine, that BSD could handle it.

[vosdev (Daniel)]

Do you have a list of machine types I could try? I was not able to run the qemu binary from the snap directory and could not chroot into the snap either

[stgraber (Stéphane Graber)]

I suspect the list of options is likely:

microvm              microvm (i386)
pc                   Standard PC (i440FX + PIIX, 1996) (alias of pc-i440fx-6.2)
pc-i440fx-6.2        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-6.1        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-6.0        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-5.2        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-5.1        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-5.0        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-4.2        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-4.1        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-4.0        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-3.1        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-3.0        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.9        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.8        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.7        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.6        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.5        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.4        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.3        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.2        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.12       Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.11       Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.10       Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.1        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.0        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.7        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.6        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.5        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.4        Standard PC (i440FX + PIIX, 1996)
q35                  Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-6.2)
pc-q35-6.2           Standard PC (Q35 + ICH9, 2009)
pc-q35-6.1           Standard PC (Q35 + ICH9, 2009)
pc-q35-6.0           Standard PC (Q35 + ICH9, 2009)
pc-q35-5.2           Standard PC (Q35 + ICH9, 2009)
pc-q35-5.1           Standard PC (Q35 + ICH9, 2009)
pc-q35-5.0           Standard PC (Q35 + ICH9, 2009)
pc-q35-4.2           Standard PC (Q35 + ICH9, 2009)
pc-q35-4.1           Standard PC (Q35 + ICH9, 2009)
pc-q35-4.0.1         Standard PC (Q35 + ICH9, 2009)
pc-q35-4.0           Standard PC (Q35 + ICH9, 2009)
pc-q35-3.1           Standard PC (Q35 + ICH9, 2009)
pc-q35-3.0           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.9           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.12          Standard PC (Q35 + ICH9, 2009)
pc-q35-2.11          Standard PC (Q35 + ICH9, 2009)
pc-q35-2.10          Standard PC (Q35 + ICH9, 2009)
isapc                ISA-only PC
none                 empty machine
x-remote             Experimental remote machine

[delmar (D Delmar Davis)]

A while back I had this solved in the #lxd channel.

The issue with freebsd 13.1 is with some escoteric feature that was altered in somewhere early lxd5 (pv_passthrough?) .

Try shutting down the VM and adding migration.stateful: “true” to the instances config.

Nice to see others running the BSDs. Someday the lxd VM agent will wind up there hint hint

[stgraber (Stéphane Graber)]

Getting the agent should be pretty straightforward.
You’ll need to build it by hand rather than running the binary available in the 9p share, but there shouldn’t be much Linux-specific logic in there.

[delmar (D Delmar Davis)]

FWIW I am not super happy with this as a solution since it’s buried in code that is moving.
I will try to find the #lxd thread with the specifics that led to this solution in case this changes again.

[stgraber (Stéphane Graber)]

The real fix should be to have the BSD kernel not crash

[delmar (D Delmar Davis)]

#lxc 11:18 < feurig> So am trying an experiment at work. Last year I ran up a freebsd 12.2 box so I could interview for my current job. I upgraded the vm to 13.0 with no problems and
then I upgraded the underlying lxd from 4 to 5.2 also with no problems.
11:20 < feurig> When I tried a fresh install with the currently supported freebsd it cores. When I upgraded my working vm to 13.1 it also cores.
11:21 < feurig> So the kernel in 13.1 isnt compatible with the current vm. (though we are running 13.1 on qemu kvms outside of lxd). And freebsd prolly doesnt care because they are
providing virtual machine images.
11:22 < feurig> My quandry is which would be more painful.
11:22 < feurig> 1. Working with the freebsd community to figure out what they broke or how to install with the old kernel.
11:22 < feurig> 2. Working with the lxd community to figure out how to unbundle the vmdk that freebsd is providing…
11:22 < feurig> any thoughts?

[delmar (D Delmar Davis)]

This is a working vm config. (boot.priority altered post install).

I am more concerned at subtle changes in lxd effecting running systems than I am with fixing the BSD kernel.

lxc config show kif
architecture: x86_64
config:
  migration.stateful: "true"
  raw.apparmor: /install/** rwk,
  raw.qemu: -boot menu=on -machine pc-q35-2.6
  security.secureboot: "false"
  volatile.cloud-init.instance-id: cace7bcf-023f-4a52-81eb-cedfc5ffe21f
  volatile.eth0.host_name: tapb7987e3e
  volatile.eth0.hwaddr: xx:xx:xx:xx:xx:xx
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: d3d7763d-0fcc-41a4-897a-dfc9ca7d4535
  volatile.vsock_id: "11"
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  install:
    boot.priority: "0"
    source: /install/FreeBSD-13.1-RELEASE-amd64-dvd1.iso
    type: disk
  root:
    boot.priority: "10"
    path: /
    pool: local
    size: 80GB
    size.state: 16GB
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: "Working VM configuration for Freebsde on LXD"

[WereCatf]

Oh, thank God! I was just trying to get OPNsense 22.7 and latest snapshot of pfSense running and I couldn’t make anything work until you mentioned having to add migration.stateful=true

I actually found that just enabling stateful migration was enough; the raw.qemu-settings or raw.apparmor-stuff weren’t needed, the VMs boot up just fine without.

[tomp (Thomas Parrott)]

We have a video on running FreeBSD now btw:

https://www.youtube.com/watch?v=OeU2SUKV5sQ

[sdeziel (Simon Deziel)]

I think you were referring to this video:

https://www.youtube.com/watch?v=OeU2SUKV5sQ

[tomp (Thomas Parrott)]

Yep its the same one I linked to

[sdeziel (Simon Deziel)]

Oh, for some reason I can’t see your link

[tomp (Thomas Parrott)]

This post seems like it could be useful to people trying to use BSD VMs too:

What qemu-setting does migration.stateful set?

I actually figured it out. Instead of migration.stateful, you can use raw.qemu: -cpu host in the virtual-machine’s configuration and be able to run more modern BSD-kernels while also letting you passthrough physical devices.

[WereCatf]

I was thinking I’d just combine this information into a brief tutorial in the tutorials-section soonish. I’m testing a few things first.

[tomp (Thomas Parrott)]

Here’s the tutorial

Run FreeBSD 13.1 / OPNsense 22.7 / pfSense 2.7.0 (and newer?) under LXD VM

Premise: Currently, starting with FreeBSD 13.1, the kernel throws a panic at boot with the default settings under LXD. This also prevents distributions using recent FreeBSD kernels from booting as well, including OPNsense 22.7 and pfSense 2.7.0. The issue appears to stem from the fact that by default, LXD tells QEMU to passthrough all supported Hyper-V enlightenments to the virtual-machines and that, presumably, causes FreeBSD’s kernel to interpret the CPU wrong. Solution: You can prevent LX…