Hi- I want to setup a server to host a mix of incus containers and virtual machines. My understanding is that ZFS is the preferred storage option and I was planning on partitioning a 500G NVME drive 100G for root and 400 for ZFS storage.
Should I partition the 100G with BTRFS or EXT4 is my basic question?
I was thinking either debian or ubuntu for root.
My understanding is ZFS is generally fine for VMs, however, I want to host a Podman virtual machine for rootless containers and my understanding is Zfs storage may not be the best for this use case, but this may of been fixed with Zfs 2.2.2. Even is this the case, ZFS 2.2.2 appears not readily available for the less tech savvy.
What storage suggestions for a podman virtual machine? I am thinking whatever is best, I will just go with that for root partition with the dir storage driver.
Any other suggestions or comments would be appreciated.
Given Incus won’t really have to deal with the 100G for your root filesystem, it doesn’t really matter. btrfs is more featureful but ext4 is usually faster and in some cases, more reliable.
Personally I tend to go with ext4 for root, but most of my machines also mostly have a throw-away root in that it doesn’t really hold any critical data to begin with.
ZFS is the most featureful of our storage drivers and works great for both containers and virtual machines.
Thanks Stephane- Should I run the latest version of zfs 2.2.3? I thought you did a video that dockers layered file system was not the best for zfs. I assumed this would also be an issue with podman which was in my original post. I plan to run a lot of OCI containers and just want it as optimized as possible. Looks forward to incus maybe natively supporting OCI directly.
ZFS 2.2 is nice because it supports VFS idmap so you get faster container creation and no need for dealing with the whole shifting business (even though that’s mostly hidden away by Incus anyways). But older ZFS will still do fine for this particular case.
Thanks again Stephane for taking the time to respond.
So with a 500G NVME will do the following:
Create 50G Root EXT4 Partition for either Debian or Ubuntu
Create 450G Incus ZFS Storage Pool and carve out 50G Ext4 block device (as you indicated above) so my Docker Containers can live in Ext4 environment
Create ZFS volume for docker data (I assume there is no issue with this)
This should provide a good environment to run a combinations of Incus/Docker containers and virtual machines. I would plan on taking a snapshot of the ZFS storage pool to another SSD device.
Is the EXT4 root partition completely throw away with this setup?
Do I also need to backup the /var/lib/incus & /etc/subuid and `/etc/subgid files for the database, profiles and other settings?
I want to easily restore the Incus server if I have a disk failure.
Mostly, you’ll still have the database and some files in /var/lib/incus that you may want to backup, but pretty much everything could be re-generated even in there through incus admin recover using just the ZFS data.
/var/lib/incus would be good. It’s also usually cheap and easy to backup the whole of /etc so you may as well since that will include all the OS configuration.
Hi I am trying this setup with latest version of incus, I created a storage pool, and a zfs volume just for docker. I attached the storage to my virtual machine. I get the following errors:
failed to start daemon: error while opening volume store metadata database
failed to start daemon: error while opening volume store metadata database (/var/lib/docker/volumes/metadata.db): no such device
I did test with a container, and it worked perfectly. Are these instructions only for containers? I also did a quick test without creating the volume just a simple vm and it worked as well. Finally I was also trying to do it this way as I wanted to carved out/reserve the space for that vm.
For a VM, you don’t need the extra volume. In fact in your case that extra volume has made /var/lib/docker use virtiofs (a kind of network filesystem) which is likely causing this error.
For VMs I’d recommend just not adding an extra volume, the VM disk image uses ext4 anyway.
If you need a separate disk to segment things, then you’ll need to make it with incus storage volume create --type=block POOL NAME size=SIZE which will create a block device (disk) rather than a filesystem.
When you attach that to a VM, it will show up as a disk, which you can then format and mount where you want the normal way (/etc/fstab).
I kept only the rootfs and rootfs snapshots on btrfs. /var/log, /var/lib/incus are zfs datasets
It works quite well*
btrfs has good performance when you stay away from quotas.
The ability to boot from read only snapshots is quite good on a testing / homelab environment.
not production ready, you can’t mount degraded filesystems, so if one disks fails you’ll be unable to boot without manual intervention.
My rootfs itself is mostly throw away, except from postfix and smartd configuration.
If I could get email notifications when a drive reports an error incus would be 100% standalone for me
