Setting up incus with ZFS properly

Hello! I was looking into incus as a database host. I also want to use ZFS, though I don’t have a lot of experience with it yet. I have a few questions about the incus + ZFS stack on ubuntu:

  1. I used a loop file for initial testing. Normally one of the main things that makes ZFS perform poorly on SSDs is setting the ashift to a bad default value. What is the default ashift value here?
  2. What is the most straightforward approach to install ZFS in a way that incus can use? No requirement for niceties like a zfs root folder. Should I be tuning it with zfs related tools separately from the settings via the incus cli?

Hi @saolof,
It does not matter how to bind storage to incus, but it does matter which database system is using, cos every db recordsize is different, i suppose. Here is a fine document that can you examine.
https://bun.uptrace.dev/postgres/tuning-zfs-aws-ebs.html#alignment-shift
Regards.

Not really relevant on a loop file since ZFS doesn’t get to control alignment with the physical disk in that scenario.

The best is when you have dedicated disks or at least partitions and tell Incus to use that. You then get a disk or partition backed pool. Recent Incus can also setup mirroring/raidz for you directly if you want.

Otherwise pre-creating a zpool or a dedicated dataset on a shared zpool and telling Incus to use that also works fine.

If you’re not ready to commit yourself 100% to ZFS, then creating a zpool inside an LVM logical volume works fine, and is easy to grow if necessary. The correct ashift should be detected automatically. (Note that ashift and recordsize are two completely different things)

If you have two physical disks, then you can create two logical volumes but one in each physical disk, and use these for ZFS mirroring. This works really well because ZFS can detect and fix data corruption when it’s doing its own mirroring. Don’t use mdraid or dmraid/LVM mirroring underneath ZFS as you’ll lose this benefit.

What is the most straightforward approach to install ZFS in a way that incus can use? No requirement for niceties like a zfs root folder.

I suggest you make your root filesystem using ext4 in one logical volume, and create another logical volume for the zpool. Leave some free space in the physical volume and then you can grow either volume later should you feel the need.

If you don’t want to dedicate the entire zpool to incus, you don’t have to. For example if the pool is called “foo” then you can create a dataset like “foo/incus” and give this to incus to use.

Hello,
As per me you can install ZFS for use with Incus sudo apt install zfsutils-linux. Then create a ZFS pool using zpool create, setting the ashift value if needed. After this configure Incus to use the ZFS pool via the Incus CLI. You may need to fine-tune ZFS settings separately using ZFS tools for optimal performance