I’ve been running a fairly simple system for quite a few years now, consisting of a 4-5 LXC containers with manually configured bridges, dnsmasq and nftables rules (including some layer 2 filtering). Each container is running unprivileged, confined by the generated apparmor profiles. Centralised logging is achieved by means of rsyslog running on both host and container, connected together via UNIX sockets. I use btrfs as the backingstore and btrbk does snapshots on a nightly basis. Thus I have a single box acting as my router, mail server, database server and web server. I guess they call it a ‘homelab’ these days.
But now I’m interested in Incus’ OCI support for sake of running zigbee2mqtt, Music Assistant and possibly even Home Assistant (though I might put the latter in an LXC container and run it via nerdctl and containerd if I can’t otherwise make it work properly without compromising security)…
Will Incus stay out of the way and let me manage the bridges and dnsmasq? Will I need to make huge changes to my nftables setup? If Incus just sets up a couple of tables, I imagine that I can just work around them. I briefly dabbed with LXD a long time ago, but came to the conclusion that it was overkill for a few containers on a single machine — KISS and all that.
for a few unprivileged boxes on one host, incus is fine. it wont force managed bridges on u.
-network type=bridge w/ bridge.external_interfaces, or unmanaged pointed at ur existing bridge, keeps dnsmasq/nftables yours
-storage btrfs driver works, leave the datasets alone, btrbk can stay on the host
-oci is native so zigbee2mqtt / music assistant can be incus launch without nesting docker just for that
-apparmor profiles still get generated per instance
u dont have to rip the old lxc stack day one. stand up incus next to it, move one container, see if the nft/bridge bits stay out of the way.
That’s great. Thanks for the comprehensive answer! I’ll probably end up nesting containerd and a bluetooth stack inside lxc for Home Assistant. I don’t think I trust HA without that extra layer of isolation, given the privileges it requires and the sprawling nature of the code base. But yes, running everything directly from Incus would be great. I find Home Assistant opinionated enough without also adding something like Proxmox to the mix! The Incus ecosystem seems to have something for everyone without being an unruly leviathan. The name was an inspired choice too.
Yes, I’d belatedly noticed you’d been working on a compose solution last night. Looks great! I suspect I will continue to use lxc for stuff like lamp and mail stacks because it is simple, elegant and easier to trust as a result. But where OCI presents a much simpler alternative to building from source and managing complex dependencies, I may yet find myself looking into using incus-compose. I built a minimal sysvinit-based Debian image that comes to around 120MB uncompressed last night that will further reduce the footprint of my lxc containers… Too many years of running servers with memory constraints.
