How to set volblocksize for a VM root on ZFS?

Incus
1

Recently I’ve discovered that the default volblocksize for zvols is 16KB. I want to run my VMs on zvols with volblocksize=128k. There’s an open issue in ZFS tracker to set volblocksize on a dataset level so it would be inherited by zvols, but until it’s implemented the size must be set in the software (incus). So I did some investigation and couldn’t find a way to set volblocksize for the root disks of VMs.

There’s a way to set the blocksize on the storage pool: incus storage set default volume.zfs.blocksize 131072. This makes all the new volumes get the proper value (128k). When a VM is started incus creates a zvol for the root disk in the same dataset, so I expected this value to apply to the root disks as well, alas it doesn’t.

I tried setting this value in the default profile: incus profile device set default root initial.zfs.blocksize=131072. This doesn’t work either: incus accepts this command, updates the profile, but this value is ignored for the new VMs.

So: is there a way to set the blocksize not only for volumes created by incus storage volume create, but also for automatically created root devices?

2

I’d have expected the initial.zfs.blocksize trick to work, can you check incus storage volume show default virtual-machine/NAME to see what ends up being set on the volume?

3

Did more testing: initial.zfs.blocksize doesn’t work. What works though is volume.zfs.blocksize for a storage, but there’s a trick: it must be set before an image is pulled. So I believe what happens is:

  1. When Incus creates a VM, it makes a quick copy of an image (existing zvol)
  2. The copy inherits volblocksize of the image and it’s not possible to override it via a config option

So, to make a VM use another volblocksize, you need to:

  1. Remove an existing image that was created with an old blocksize: incus image rm ...

  2. Set the desired blocksize: incus storage set default volume.zfs.blocksize 131072

  3. Create a VM: incus launch images:debian/13 test --vm

    So Incus would pull an image into a new zvol with the specified blocksize, and all the new VMs will inherit this value.

So this basically resolves my issue, I now can create VMs as I want them to be.

As for the initial.zfs.blocksize, found this in https://linuxcontainers.org/incus/docs/main/reference/devices_disk:

Note that you cannot use initial volume configurations with custom volume options or to set the volume’s size.

So it’s probably expected.

UPD: Here’s how it looks like: one old image and one new

~ ❯ zfs list -r -o name,volblocksize data/incus/images
NAME                                                                                      VOLBLOCK
data/incus/images                                                                                -
data/incus/images/c20de1cbe2327627fe41db7833910f7b6c74c246528cbe9a81bff01ea9489b92               -
data/incus/images/c20de1cbe2327627fe41db7833910f7b6c74c246528cbe9a81bff01ea9489b92.block       16K
data/incus/images/cd8c3e41e84dd5105c9b7bfae24ea96a2fcfb8883c93007decc113038ebee2bc               -
data/incus/images/f5b430ec3ce7e741c3806c1d3e70eee842580ba892970084eefbe7861f3815e4               -
data/incus/images/f5b430ec3ce7e741c3806c1d3e70eee842580ba892970084eefbe7861f3815e4.block      128K
4

Ah yeah, that actually makes a lot of sense.

Because the instances are clones of the image, the blocksize will only apply on initial image unpack into ZFS.

5

So that makes it only really viable as a volume option on the pool, unless you have ZFS clones disabled on the pool (but then you’ll get much higher disk usage).