I am doing a PoC for Incus at work and I’m thrilled. Incus addresses many of the issues I have with the other VM/system containers products I’ve used- in the way that I would address them if I had proper time.
(I’m so thrilled, I ordered some extra RAM to add Incus to my personal infrastructure too.)
I’m wondering if there’s any “suggested” environment that is recommended. For work, we want a bare metal virtualization lab- we’re using cheap Hetzner dedicated servers for this purpose. But I’m seeing some crashes and issues, because we run quite complex workloads- basically the Android emulator, sometimes nested in a Docker container, running on an Incus VM.
My POC setup is:
Debian Stable
The Debian Backports version of Incus
The dir storage backend.
I already want to replace dir with BTRFS, LVM, or ZFS, to get better snapshotting, but I wonder if I would benefit a lot from running another base distribution with a more modern kernel and/or running newer versions of Incus. (Or maybe I just need to adjust the hardware on the VMs I create…).
I kinda like running LTS-y distros, although for work, this would all be disposable environments, so I wouldn’t mind running something more bleeding edge (e.g. Debian Testing, Fedora, etc.) if that’s going to make things work better.
(I love ZFS, so something that supports ZFS well would be awesome.)
What is everyone using? Can we know what underlying OSes are being used for development, for example?
The distro shouldn’t really matter.
For kernel, it may matter but Debian 12’s kernel isn’t particularly old.
If you want to try something newer you can check out GitHub - zabbly/linux: Linux kernel builds
All my own clusters are either on Ubuntu 22.04 or Debian 12 at this stage, but again, the distro itself doesn’t really matter.
ZFS is definitely the most flexible for local storage. It’s not going to make things faster than ext4 but you will get support for snapshots, better backups and a ton more option per instance. The most important being the ability to set proper quotas on things
If I find some more time, I might try to find an easy way to reproduce our issues with the Android emulator and perhaps report some bug. (I’d like also to try with a container instead of a VM.) And of course, retest with the upstream kernel/Incus packages.
Quotas actually work with ZFS and aren’t trivially bypassable like with btrfs.
ZFS also has proper native support for providing block devices as we use for VMs, btrfs instead requires the VM to be stored as a file on the filesystem leading to very poor performance.
ZFS also has control of block/record size on a per-instance basis which allows for much better performance for databases by lining up the on-disk block/record size with what the database tends to write.
I’m pretty sure there are more, especially around delegation features, but quotas and VM handling are really the two big ones.
