A fun project: microvms with Incus
I saw this blog post for microvms in Proxmox, and it got me thinking, “Couldn’t I do the same in Incus?” So I decided to mess around and see if I could end up with a profile I could make, to make use of qemu’s microvm feature in Incus. It turns out, it’s entirely possible to get it working, and it works better than I could have imagined.
What’s a microvm?
Here is QEMU’s own description:
microvm is a machine type inspired by Firecracker and constructed after its machine model. It’s a minimalist machine type without PCI nor ACPI support, designed for short-lived guests. microvm also establishes a baseline for benchmarking and optimizing both QEMU and guest operating systems, since it is optimized for both boot time and footprint.
A regular Incus VM is a full emulated machine: a q35 chipset, a PCIe topology, ACPI, and UEFI firmware (OVMF) that runs a bootloader off the guest disk, which then loads the guest’s own kernel and initramfs. It behaves like a physical PC. A microvm throws nearly all of that away. There is no firmware in the boot path at all, so QEMU is handed a kernel directly and jumps straight into it. The result sits much closer to a container in overhead, while still being a real, isolated VM with its own kernel.
One thing surprised me, and it is worth clearing up because QEMU’s wording is a little misleading. The definition says “without PCI nor ACPI,” but that describes the bare default (pure virtio-mmio, the Firecracker-like model). The microvm machine type also has opt-in pcie=on and acpi=on switches. I turn both on. That is deliberate: PCI is what lets Incus hot-plug a normal NIC into the guest, and it is also what made the SR-IOV experiment below work. So you get to choose where on the minimalism spectrum you want to sit.
The catch: you supply the kernel
Because there is no firmware and no bootloader, the guest’s own kernel never runs. You hand QEMU a kernel with -kernel, and that kernel has to be able to mount any guest’s root filesystem and bring up the Incus agent entirely on its own.
So I built a small, monolithic kernel (CONFIG_MODULES=n, everything compiled in, no /lib/modules needed in the guest at all) tuned for my CPU. The rule is that everything the guest could need has to be built in (=y), because there is no initramfs to load modules from:
- virtio blk / net / scsi / console / balloon / rng, plus
VIRTIO_PCI - 9p + virtio-9p + vsock + virtio-vsock (this is what the Incus agent rides on; miss it and the VM boots fine but the agent never appears)
- ext4, vfat, overlayfs, virtiofs
- the netfilter stack, TCP BBR, and so on
BPF_SYSCALL(modern systemd wants it)- and, just to see if it would work, the
iavfdriver for Intel SR-IOV virtual functions
Stripped down from a distro defconfig it lands around 12 MiB and boots a guest to userspace in roughly a quarter of a second.
The profile
This is where Incus really shines. raw.qemu.scriptlet lets you rewrite the QEMU configuration in Starlark before the VM launches. I use it to switch the machine type to microvm and drop the firmware bits Incus adds for q35:
def qemu_hook(instance, stage):
if stage == "config":
out = []
for s in get_qemu_conf():
n, e = s["name"], s["entries"]
if n == "machine":
e["type"] = "microvm"
e["acpi"] = "on"; e["pcie"] = "on"; e["rtc"] = "on"
elif n == "drive" and e.get("if") == "pflash":
continue # no OVMF
elif n == "global" and e.get("driver") == "ICH9-LPC":
continue # q35 southbridge, gone
out.append(s)
set_qemu_conf(out)
The scriptlet can rewrite the machine config, but it cannot add -kernel. That goes in plain raw.qemu. Here is the whole profile:
config:
security.secureboot: "false"
limits.cpu: sockets=1,cores=4,threads=1
limits.memory: 2GiB
raw.qemu: -kernel /path/to/microvm-kernel -append "root=/dev/vda2 rootwait rw console=ttyS0"
raw.qemu.scriptlet: |-
... the Starlark above ...
devices:
eth0:
type: nic
nictype: bridged
parent: br0
root:
type: disk
path: /
pool: default
io.bus: virtio-blk # makes the disk show up as /dev/vda
And then it is just:
incus launch images:debian/13 test --vm -p microvm
The gotchas
None of these were obvious, so I am writing them down for anyone who wants to try this:
-
CPU hotplug. Incus always asks QEMU about hot-pluggable CPUs, and microvm does not support that, so the VM dies on start. The fix is to set
limits.cpuas a topology (sockets=1,cores=4,threads=1) rather than a plain number. A fixed topology disables hotplug and the error goes away. -
Secure Boot. Incus refuses to create the VM (“incompatible with secureboot”) before the scriptlet ever runs. Since there is no OVMF here anyway,
security.secureboot=falseis the correct setting, not a compromise. -
ro vs rw, the sneaky one. With
root=... roin the append, Alpine and Debian were fine, but Arch came up with no networking and no DNS. The cause: Arch’s cloud image ships with no/etc/machine-idand relies on systemd generating one on first boot. At that moment/etcis still read-only, so systemd cannot write it, and with no machine-id it cannot build a DHCP DUID, so systemd-networkd fails the whole link. A normal VM dodges this because its initramfs has root writable in time. Bootingrwfixes it cleanly, and Arch then generates its own machine-id on first boot like it should. -
BPF_SYSCALL. Without it, systemd logs
Unable to load sysctl monitor BPF program: Function not implemented. Not fatal, but build it in to keep modern systemd happy.
Does one profile really boot anything?
That was the real question: one profile, or one profile per image? It turns out to be genuinely one profile. With the same microvm profile I launched Alpine, Arch, and Debian/trixie, all unmodified, and all three boot, get an IP, and resolve DNS with no per-image tweaks. Even better, the incus agent works just like a normal VM. The only assumptions are an ext4 root on partition 2 (every Incus image I tried uses that) and the matching filesystem built into the kernel. Because you bypass the image’s own initramfs, anything that depends on it (LUKS, LVM-on-root) will not work, but ordinary cloud images just go.
The numbers
Same Alpine image, same 1 GiB of RAM, idle just after boot:
| microvm | normal q35 + OVMF | |
|---|---|---|
| host RAM (qemu Pss) | ~196 MiB | ~379 MiB |
Boot time, measured as incus start to the agent answering, warm restarts, same Alpine image:
| microvm (our kernel, direct boot) | q35 (OVMF + image kernel + initramfs) | |
|---|---|---|
| start to agent | ~5 s | ~8 s |
The memory saving is the headline here: roughly half the host RAM for the same guest. With the high cost of RAM lately, this matters, and it matters more the more guests you pack onto a box.
The boot speedup is real too, but more modest, around three seconds. It is worth being precise about where it comes from, because it is not just skipping the bootloader. A normal VM spends time in the OVMF firmware, then the bootloader, then the kernel’s own initramfs, all before userspace even starts. The microvm skips that entire chain and runs a smaller kernel that initializes in about a quarter of a second. Once the distro’s userspace and the Incus agent come up, the two paths are identical, which is why the gap is a few seconds rather than tenfold. For a single VM that is minor; for a fleet of short-lived guests it adds up, and the RAM difference adds up a good deal faster.
Bonus: SR-IOV works too
Since I had built the iavf driver into the kernel, I tried attaching an SR-IOV virtual function from a physical NIC to one of these microvms, with a VLAN set on the VF. It just worked. The host hands the VF through, the guest drives the passed-through NIC natively with iavf, and the link comes up tagged on the correct VLAN. So these are not just toys; they can have real, line-rate hardware networking.
This is also why I keep PCI enabled. I did consider the pure virtio-mmio (no-PCI) variant, which is the most minimal microvm of all. But when I actually measured it, the difference was tiny: roughly 2 MiB less host RAM and only a couple dozen milliseconds faster to boot. That is not worth giving up PCI for, because PCI is what makes device passthrough and NIC hot-plug possible in the first place. So I leave pcie=on and get the passthrough abilities essentially for free.
What I have not tried (and probably will not work)
To be clear about scope: I only ran this on a single host in a homelab. I have not touched any of the cluster-oriented features, and I would expect most of them not to work as-is. Live migration, cluster scheduling and evacuation, and anything that assumes a standard machine type or a stable firmware and CPU model across hosts all seem unlikely to behave. Snapshots and config copies of the instance should be fine, but I have not stress-tested those either. Treat this as “fun and genuinely useful on one box,” not “production cluster-ready.”
I would love to see this officially supported
What strikes me most is how little it took: a small kernel and a short scriptlet, and you end up with what feels like a real middle ground between a container and a VM. A true, isolated guest with its own kernel, but at close-to-container overhead and boot time. I would love to see microvm become a first-class, supported option in Incus, because from the outside it does not look like it would be hard to add.
And here is the best part: none of this requires patching or modifying Incus at all. Everything above is just configuration and a scriptlet on a stock Incus. So even though it is not official, it is already entirely available to anyone who wants to do it today.
What I plan to do next
I’m currently working on a way to convert OCI images and container rootfs to microvms that work in Incus. If I’m able to accomplish all that, I’m considering proposing it as a new feature for future versions of Incus, if that’s a direction the project is interested in going.
Happy to share more info if anyone has any questions or needs help.
