Hello,
first of all, thank you for Incus and your dedication to the project! What a great software.
Description
If it is more (e.g. 4 GB) the FreeBSD VM throws the following error:
Reducing the RAM to 2 GB or below, the FreeBSD VMs function as expected. I tried it with two fresh debian trixie bare metal installations on the same host with fresh OPNSense and PFSense installations. When using LTS and more than 2 GB RAM not problem, when using 6.14 the above crash happens.
Incus Config / Steps to reproduce
Installed a fresh and clean debian trixie RC1; one time with Incus LTS 6.0.4 and one time with Incus 6.14 (same host, bare metal) follwoing the exact steps form GitHub - zabbly/incus: Incus package repository
I did not upgrade or copy things or vm; it was two complete clean new Debian OS installs and vm creations to test the difference between Incus versions.
Create a new VM with exactly the same config (see below) used on 6.0.4 and then on 6.14 with OPNsense 25.1 ISO (same behavior with PFSense 2.8.0)
architecture: x86_64
config:
limits.cpu: "2"
limits.memory: 4GiB
security.secureboot: "false"
volatile.cloud-init.instance-id: cf3ee5e9-ea25-4318-b165-d21210862955
volatile.eth0.host_name: tap27ebefbf
volatile.eth0.hwaddr: 10:66:6a:db:f4:e3
volatile.last_state.power: RUNNING
volatile.uuid: cf713c67-9bca-4324-90c1-2a6104872098
volatile.uuid.generation: cf713c67-9bca-4324-90c1-2a6104872098
volatile.vm.definition: pc-q35-9.0
volatile.vsock_id: "1118946314"
devices:
iso-volume:
boot.priority: "10"
pool: default
source: OPNsense-25.1-dvd-amd64.iso
type: disk
root:
path: /
pool: default
size: 32GiB
type: disk
ephemeral: false
profiles:
- default
stateful: false
description: test
Start the VM. The error from section “Description” occurs on Incus 6.14 with more than 2 GB RAM, not on Incus LTS 6.0.4.
Any Ideas what could cause this?
stgraber
June 29, 2025, 3:47pm
2
Can you show the content of /run/incus/NAME/qemu.conf?
Sure. Here the file content from 6.14:
# Machine
[machine]
graphics = "off"
accel = "kvm"
usb = "off"
type = "q35"
[global]
driver = "ICH9-LPC"
property = "disable_s3"
value = "1"
[global]
value = "0"
driver = "ICH9-LPC"
property = "disable_s4"
[boot-opts]
strict = "on"
# Memory
[memory]
slots = "8"
size = "4096M"
maxmem = "1048576M"
# CPU
[smp-opts]
cpus = "1"
maxcpus = "32"
[object "mem0"]
qom-type = "memory-backend-memfd"
size = "4096M"
share = "on"
[numa]
type = "node"
nodeid = "0"
memdev = "mem0"
# Firmware (read only)
[drive]
file = "/opt/incus/share/qemu/OVMF_CODE.4MB.fd"
if = "pflash"
format = "raw"
unit = "0"
readonly = "on"
# Firmware settings (writable)
[drive]
if = "pflash"
format = "raw"
unit = "1"
file = "/dev/fd/4"
# Qemu control
[chardev "monitor"]
path = "/run/incus/opnsense/qemu.monitor"
backend = "socket"
server = "on"
wait = "off"
[mon]
chardev = "monitor"
mode = "control"
# Console
[chardev "console"]
size = "1048576"
backend = "ringbuf"
# VM core info driver
[device "qemu_vmcoreinfo"]
driver = "vmcoreinfo"
[device "qemu_pcie0"]
chassis = "0"
multifunction = "on"
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.0"
# Balloon driver
[device "qemu_balloon"]
addr = "00.0"
multifunction = "on"
driver = "virtio-balloon-pci"
bus = "qemu_pcie0"
# Random number generator
[object "qemu_rng"]
qom-type = "rng-random"
filename = "/dev/urandom"
[device "dev-qemu_rng"]
driver = "virtio-rng-pci"
bus = "qemu_pcie0"
addr = "00.1"
rng = "qemu_rng"
# Input
[device "qemu_keyboard"]
bus = "qemu_pcie0"
addr = "00.2"
driver = "virtio-keyboard-pci"
# Input
[device "qemu_tablet"]
addr = "00.3"
driver = "virtio-tablet-pci"
bus = "qemu_pcie0"
# Vsock
[device "qemu_vsock"]
driver = "vhost-vsock-pci"
bus = "qemu_pcie0"
addr = "00.4"
guest-cid = "2430697077"
vhostfd = "3"
# Virtual serial bus
[device "dev-qemu_serial"]
bus = "qemu_pcie0"
addr = "00.5"
driver = "virtio-serial-pci"
# Serial identifier
[chardev "qemu_serial-chardev"]
size = "16B"
backend = "ringbuf"
[device "qemu_serial"]
name = "org.linuxcontainers.incus"
bus = "dev-qemu_serial.0"
chardev = "qemu_serial-chardev"
driver = "virtserialport"
[device "qemu_serial_legacy"]
driver = "virtserialport"
name = "org.linuxcontainers.lxd"
bus = "dev-qemu_serial.0"
# Spice agent
[chardev "qemu_spice-chardev"]
backend = "spicevmc"
name = "vdagent"
[device "qemu_spice"]
driver = "virtserialport"
name = "com.redhat.spice.0"
chardev = "qemu_spice-chardev"
bus = "dev-qemu_serial.0"
# Spice folder
[chardev "qemu_spicedir-chardev"]
backend = "spiceport"
name = "org.spice-space.webdav.0"
[device "qemu_spicedir"]
driver = "virtserialport"
name = "org.spice-space.webdav.0"
chardev = "qemu_spicedir-chardev"
bus = "dev-qemu_serial.0"
# USB controller
[device "qemu_usb"]
bus = "qemu_pcie0"
addr = "00.6"
p2 = "8"
p3 = "8"
driver = "qemu-xhci"
[chardev "qemu_spice-usb-chardev1"]
backend = "spicevmc"
name = "usbredir"
[device "qemu_spice-usb1"]
driver = "usb-redir"
chardev = "qemu_spice-usb-chardev1"
[chardev "qemu_spice-usb-chardev2"]
backend = "spicevmc"
name = "usbredir"
[device "qemu_spice-usb2"]
driver = "usb-redir"
chardev = "qemu_spice-usb-chardev2"
[chardev "qemu_spice-usb-chardev3"]
backend = "spicevmc"
name = "usbredir"
[device "qemu_spice-usb3"]
driver = "usb-redir"
chardev = "qemu_spice-usb-chardev3"
[device "qemu_pcie1"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.1"
chassis = "1"
# SCSI controller
[device "qemu_scsi"]
driver = "virtio-scsi-pci"
bus = "qemu_pcie1"
addr = "00.0"
[device "qemu_pcie2"]
chassis = "2"
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.2"
# Shared config drive (9p)
[fsdev "qemu_config"]
security_model = "none"
readonly = "on"
path = "/var/lib/incus/devices/opnsense/config.mount"
fsdriver = "local"
[device "dev-qemu_config-drive-9p"]
driver = "virtio-9p-pci"
bus = "qemu_pcie2"
addr = "00.0"
multifunction = "on"
mount_tag = "config"
fsdev = "qemu_config"
# Shared agent drive (9p)
[fsdev "qemu_agent"]
security_model = "none"
readonly = "on"
path = "/opt/incus/agent/"
fsdriver = "local"
[device "dev-qemu_agent-drive-9p"]
addr = "00.1"
mount_tag = "agent"
fsdev = "qemu_agent"
driver = "virtio-9p-pci"
bus = "qemu_pcie2"
[device "qemu_pcie3"]
chassis = "3"
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.3"
# GPU
[device "qemu_gpu"]
driver = "virtio-vga"
bus = "qemu_pcie3"
addr = "00.0"
[device "qemu_pcie4"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.4"
chassis = "4"
# VM Generation ID
[device "vmgenid0"]
driver = "vmgenid"
guid = "b74f6c4a-1322-4f30-bad5-658383d979e7"
[device "qemu_pcie5"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.5"
chassis = "5"
[device "qemu_pcie6"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.6"
chassis = "6"
[device "qemu_pcie7"]
addr = "1.7"
chassis = "7"
driver = "pcie-root-port"
bus = "pcie.0"
[device "qemu_pcie8"]
multifunction = "on"
driver = "pcie-root-port"
bus = "pcie.0"
addr = "2.0"
chassis = "8"
[device "qemu_pcie9"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "2.1"
chassis = "9"
[device "qemu_pcie10"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "2.2"
chassis = "10"
[device "qemu_pcie11"]
chassis = "11"
driver = "pcie-root-port"
bus = "pcie.0"
addr = "2.3"
[device "qemu_pcie12"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "2.4"
chassis = "12"
and here from 6.0.4
# Machine
[machine]
graphics = "off"
type = "q35"
accel = "kvm"
usb = "off"
[global]
driver = "ICH9-LPC"
property = "disable_s3"
value = "1"
[global]
driver = "ICH9-LPC"
property = "disable_s4"
value = "0"
[boot-opts]
strict = "on"
# Memory
[memory]
size = "4096M"
# CPU
[smp-opts]
cpus = "1"
maxcpus = "32"
[object "mem0"]
qom-type = "memory-backend-memfd"
size = "4096M"
share = "on"
[numa]
type = "node"
nodeid = "0"
memdev = "mem0"
# Firmware (read only)
[drive]
file = "/opt/incus/share/qemu/OVMF_CODE.4MB.fd"
if = "pflash"
format = "raw"
unit = "0"
readonly = "on"
# Firmware settings (writable)
[drive]
file = "/dev/fd/4"
if = "pflash"
format = "raw"
unit = "1"
# Qemu control
[chardev "monitor"]
backend = "socket"
path = "/run/incus/opnsense/qemu.monitor"
server = "on"
wait = "off"
[mon]
chardev = "monitor"
mode = "control"
# Console
[chardev "console"]
backend = "ringbuf"
size = "1048576"
# VM core info driver
[device "qemu_vmcoreinfo"]
driver = "vmcoreinfo"
[device "qemu_pcie0"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.0"
chassis = "0"
multifunction = "on"
# Balloon driver
[device "qemu_balloon"]
driver = "virtio-balloon-pci"
bus = "qemu_pcie0"
addr = "00.0"
multifunction = "on"
# Random number generator
[object "qemu_rng"]
qom-type = "rng-random"
filename = "/dev/urandom"
[device "dev-qemu_rng"]
driver = "virtio-rng-pci"
bus = "qemu_pcie0"
addr = "00.1"
rng = "qemu_rng"
# Input
[device "qemu_keyboard"]
driver = "virtio-keyboard-pci"
bus = "qemu_pcie0"
addr = "00.2"
# Input
[device "qemu_tablet"]
driver = "virtio-tablet-pci"
bus = "qemu_pcie0"
addr = "00.3"
# Vsock
[device "qemu_vsock"]
driver = "vhost-vsock-pci"
bus = "qemu_pcie0"
addr = "00.4"
guest-cid = "1118946314"
vhostfd = "3"
# Virtual serial bus
[device "dev-qemu_serial"]
driver = "virtio-serial-pci"
bus = "qemu_pcie0"
addr = "00.5"
# Serial identifier
[chardev "qemu_serial-chardev"]
backend = "ringbuf"
size = "16B"
[device "qemu_serial"]
driver = "virtserialport"
name = "org.linuxcontainers.incus"
chardev = "qemu_serial-chardev"
bus = "dev-qemu_serial.0"
[device "qemu_serial_legacy"]
driver = "virtserialport"
name = "org.linuxcontainers.lxd"
bus = "dev-qemu_serial.0"
# Spice agent
[chardev "qemu_spice-chardev"]
backend = "spicevmc"
name = "vdagent"
[device "qemu_spice"]
driver = "virtserialport"
name = "com.redhat.spice.0"
chardev = "qemu_spice-chardev"
bus = "dev-qemu_serial.0"
# Spice folder
[chardev "qemu_spicedir-chardev"]
backend = "spiceport"
name = "org.spice-space.webdav.0"
[device "qemu_spicedir"]
driver = "virtserialport"
name = "org.spice-space.webdav.0"
chardev = "qemu_spicedir-chardev"
bus = "dev-qemu_serial.0"
# USB controller
[device "qemu_usb"]
driver = "qemu-xhci"
bus = "qemu_pcie0"
addr = "00.6"
p2 = "8"
p3 = "8"
[chardev "qemu_spice-usb-chardev1"]
backend = "spicevmc"
name = "usbredir"
[device "qemu_spice-usb1"]
driver = "usb-redir"
chardev = "qemu_spice-usb-chardev1"
[chardev "qemu_spice-usb-chardev2"]
backend = "spicevmc"
name = "usbredir"
[device "qemu_spice-usb2"]
driver = "usb-redir"
chardev = "qemu_spice-usb-chardev2"
[chardev "qemu_spice-usb-chardev3"]
backend = "spicevmc"
name = "usbredir"
[device "qemu_spice-usb3"]
driver = "usb-redir"
chardev = "qemu_spice-usb-chardev3"
[device "qemu_pcie1"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.1"
chassis = "1"
# SCSI controller
[device "qemu_scsi"]
driver = "virtio-scsi-pci"
bus = "qemu_pcie1"
addr = "00.0"
[device "qemu_pcie2"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.2"
chassis = "2"
# Shared config drive (9p)
[fsdev "qemu_config"]
fsdriver = "local"
security_model = "none"
readonly = "on"
path = "/var/lib/incus/devices/opnsense/config.mount"
[device "dev-qemu_config-drive-9p"]
driver = "virtio-9p-pci"
bus = "qemu_pcie2"
addr = "00.0"
multifunction = "on"
mount_tag = "config"
fsdev = "qemu_config"
# Shared agent drive (9p)
[fsdev "qemu_agent"]
fsdriver = "local"
security_model = "none"
readonly = "on"
path = "/opt/incus/agent/"
[device "dev-qemu_agent-drive-9p"]
driver = "virtio-9p-pci"
bus = "qemu_pcie2"
addr = "00.1"
mount_tag = "agent"
fsdev = "qemu_agent"
[device "qemu_pcie3"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.3"
chassis = "3"
# GPU
[device "qemu_gpu"]
driver = "virtio-vga"
bus = "qemu_pcie3"
addr = "00.0"
[device "qemu_pcie4"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.4"
chassis = "4"
# VM Generation ID
[device "vmgenid0"]
driver = "vmgenid"
guid = "cf713c67-9bca-4324-90c1-2a6104872098"
[device "qemu_pcie5"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.5"
chassis = "5"
[device "qemu_pcie6"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.6"
chassis = "6"
[device "qemu_pcie7"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.7"
chassis = "7"
[device "qemu_pcie8"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "2.0"
chassis = "8"
multifunction = "on"
[device "qemu_pcie9"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "2.1"
chassis = "9"
[device "qemu_pcie10"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "2.2"
chassis = "10"
[device "qemu_pcie11"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "2.3"
chassis = "11"
[device "qemu_pcie12"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "2.4"
chassis = "12"
stgraber
June 29, 2025, 7:56pm
4
Can you try setting raw.qemu.conf to
[memory]
maxmem = "65536M`
I wonder if it’s the 1TB max hotplug memory which is tripped FreeBSD somehow.
Thanks for you fast reply. Adding a file /run/incus/NAMEOFVM/raw.qemu.conf with the content
[memory]
maxmem = "65536M"
and then stopping and starting the VM again does not help. I also aligned the values with “4096M” and tried replacing the whole raw.qemu.conf with the content of qemu.conf from the Incus 6.0.4 install.
No success
osch
June 30, 2025, 2:16am
6
This approach will fail by design as Incus rewrites the file on every start of the instance. You need to edit your instance configuration (incus config edit <instance-name>) and add the value there. You might need to add raw.qemu.conf as it might not exists.
You are right, Instance options - Incus documentation describes it very clearly.
running incus config edit VMNAME and adding the following
config:
(...)
raw.qemu.conf: |-
[memory]
maxmem = "65536M"
(...)
solves the problem
FreeBSD VMs now booting up perfectly fine with > 2 GB RAM on Incus 6.14.
Thank you for your fast support!
stgraber
June 30, 2025, 7:05pm
8
Given some of the weirdness we’ve had with calculating those memory amounts, there’s a good chance I’ll be introducing a limits.memory.hotplug to control the amount of hot-pluggable memory so we can tweak that without having to use raw.qemu.conf.
