Over the past week, I rebuilt and open-sourced my personal multi-host infrastructure repository, r3x. It manages baremetal workstations, Incus virtual machines, and Incus LXC development containers using NixOS.
Since Incus is the backbone of all my container and VM workloads, I wanted to share a few patterns, lessons, and one specific nftables trick that solved a long-standing firewall headache when running Incus on NixOS hosts.
1. The NixOS Firewall + Incus Bridge Dilemma (and an nftables trick)
If you run Incus on a NixOS host, you’ve probably run into the firewall battle:
- Incus manages its own firewall rules under
table inet incusand dynamically tracks all active bridge interfaces in a set called@bridges. - NixOS, by default, has its own strict firewall (
nixos-fw) that drops unmatched incoming and forwarded packets. - In the past, people either had to explicitly declare every bridge name (e.g.,
networking.firewall.trustedInterfaces = [ "incusbr0" ]) which breaks dynamic networks, or disable the NixOS firewall entirely.
To solve this cleanly and dynamically without hardcoding interface names, I attached early prerouting and forward hooks to mark any packet belonging to an Incus-managed bridge:
networking.nftables.enable = true;
# Incus maintains `table inet incus` and dynamically populates `set bridges`.
# We attach early prerouting and forward hooks at priority -150 to mark
# packets from/to `@bridges` with 0x494e43 (ASCII "INC").
networking.nftables.tables."incus" = {
family = "inet";
content = ''
set bridges {
type ifname
}
chain incus_bridge_mark_prerouting {
type filter hook prerouting priority -150;
iifname @bridges meta mark set 0x494e43
}
chain incus_bridge_mark_forward {
type filter hook forward priority -150;
oifname @bridges meta mark set 0x494e43
}
'';
};
# NixOS firewall simply accepts any packet carrying the Incus mark
networking.firewall = {
extraInputRules = ''
meta mark 0x494e43 accept comment "trust all incus managed bridges via @bridges"
'';
extraForwardRules = ''
meta mark 0x494e43 accept comment "allow traffic for all incus managed bridges via @bridges"
'';
};
Because 0x494e43 is set based on Incus’s own dynamic @bridges set, you can create, rename, or destroy Incus bridges at runtime and the host firewall automatically trusts them without needing a nixos-rebuild switch.
2. Dual Deployment: LXC Devcontainers vs. Graphical Desktop VMs
In r3x, instances are split into distinct host templates:
devcontainer(Incus LXC): Lightweight, unprivileged container for isolated terminal development, SSH, and toolchains.desktop-vm(Incus VM): Full virtual machine running graphical Wayland desktops (like COSMIC or Niri) with QEMU integration viavirtualisation/incus-virtual-machine.nix.
A Justfile recipe automates image generation directly from our Nix expressions:
- For containers, Nix builds the squashfs rootfs and metadata tarball:
metadata=$(nix build ".#nixosConfigurations.${host}.config.system.build.metadata" --print-out-paths) squashfs=$(nix build ".#nixosConfigurations.${host}.config.system.build.squashfs" --print-out-paths) incus image import "$metadata/tarball/nixos-image-lxc-*.tar.xz" "$squashfs/nixos-lxc-image-*.squashfs" "$remote:" --alias "nixos/$host" - For VMs, Nix builds the qcow2 disk image and VM metadata tarball:
qemu_img=$(nix build ".#nixosConfigurations.${host}.config.system.build.qemuImage" --print-out-paths) incus image import "$metadata/tarball/nixos.tar.xz" "$qemu_img/nixos.qcow2" "$remote:" --alias "nixos/$host"
3. Declarative Provisioning via OpenTofu + Persistent Storage
Once the images are built and pushed to the Incus remote, OpenTofu applies the instance definitions.
To keep containers and VMs truly ephemeral while preserving SSH host identity and user data across rebuilds, each instance gets an independent incus_storage_volume:
resource "incus_storage_volume" "persist" {
count = var.persist_volume ? 1 : 0
name = "${var.host}-persist"
pool = var.storage_pool
remote = var.incus_remote
content_type = "filesystem"
# Injects persistent SSH host keys decrypted from our YubiKey Age identities
dynamic "file" {
for_each = var.persist_files
content {
target_path = file.value.target_path
source_path = lookup(file.value, "source_path", null)
mode = lookup(file.value, "mode", "0600")
}
}
}
resource "incus_instance" "instance" {
name = var.host
type = var.instance_type
image = "nixos/${var.host}"
remote = var.incus_remote
running = true
profiles = var.profiles
device {
name = "persist"
type = "disk"
properties = {
pool = var.storage_pool
source = incus_storage_volume.persist[0].name
path = "/persist"
}
}
}
This means I can destroy and recreate an Incus container at will (incus delete -f dev01), re-apply via OpenTofu, and it comes right back up with its persistent /persist storage mounted, host SSH keys intact, and zero manual setup.
4. Incus Preseed in NixOS
For the baremetal host, the entire Incus daemon, default network, and default storage pools are preseeded directly in Nix:
virtualisation.incus = {
enable = true;
package = pkgs.incus;
preseed = {
config = {
"core.https_address" = ":8443";
};
networks = [
{
name = "incusbr0";
type = "bridge";
config = {
"ipv4.address" = "10.90.190.1/24";
"ipv4.nat" = "true";
"ipv6.address" = "auto";
"ipv6.nat" = "true";
};
}
];
storage_pools = [
{
name = "default";
driver = "btrfs";
config = {
source = "/var/lib/incus/storage-pools/default";
};
}
];
};
};
The full configuration, OpenTofu definitions, and recipes are available on GitHub:
Repository: https://github.com/jochumdev/r3x
Blog Post: https://jochum.dev/en/tech/20261004-r3x-zero-trust-nixos-with-vaultix-and-den/
If anyone else is running NixOS guests or using NixOS as an Incus hypervisor, I’d love to hear how you handle network trust and persistent storage!