Recommended general storage pool

Incus
1

In my new Incus build, I wish to be able to run Docker containers inside Incus containers. In @stgraber 's excellent video on the subject, he uses brtfs as Docker doesn’t presently cooperate with ZFS. In my existing 5yo LXD install, I used LVM with thin provisioning as my storage pool. I haven’t really been able to find guidance on recommendations for general-purpose storage pools. I think I originally picked LVM/thinpool for ease of snapshotting for backups and checkpoints if I was going to do a software upgrade in a container that might go sour. It’s been great for this, but maybe brtfs is better?

  1. Use brtfs for everything - docker will work fine like in the video
  2. Use LVM for everything - docker will still work fine(?)
  3. Use LVM for most things, but create separate brtfs pool for docker-using containers

Edit: Bonus points for simplicity!

TIA,
-brmiller

2

I use ZFS for everything except podman/docker workloads.

I would recommend starting over with Incus and ZFS, then follow the directions in the video. There is another video somewhere about different storage options. ZFS seemed to win over LVM.

incus launch images:debian/12/cloud d1
incus storage create podman btrf
incus storage volume create podman tmp-btfs-volume
incus storage volume attach podman tmp-btfs-volume d1 /var/lib/containers
incus config set d1 security.nesting=true security.syscalls.intercept.mknod=true security.syscalls.intercept.setxattr=true
incus restart d1
incus exec d1 -- su -l debian

Then from inside your container:

sudo apt update
sudo apt install podman
podman run -it docker.io/ubuntu /bin/bash

All seems to work. I did not experience any performance issues. I have in the past before I learned about the special needs of these overlay filesystems.

1 Like
3

I think this is the video I was remembering.

4

Worth noting that you can now use a block-mode ZFS volume that will get you an ext4 volume from your ZFS pool.

stgraber@dakara:~$ incus launch images:ubuntu/22.04 u1
Launching u1
stgraber@dakara:~$ incus storage volume create default u1-docker zfs.block_mode=true size=10GiB
Storage volume u1-docker created
stgraber@dakara:~$ incus config device add u1 docker disk pool=default source=u1-docker path=/var/lib/docker
Device docker added to u1
stgraber@dakara:~$ incus exec u1 bash
root@u1:~# df -h
Filesystem                                       Size  Used Avail Use% Mounted on
dakara/incus/containers/u1                       670G  264M  670G   1% /
none                                             492K  4.0K  488K   1% /dev
efivarfs                                         128K   33K   91K  27% /sys/firmware/efi/efivars
tmpfs                                            100K     0  100K   0% /dev/incus
tmpfs                                            100K     0  100K   0% /dev/.incus-mounts
tmpfs                                             32G     0   32G   0% /dev/shm
tmpfs                                             13G  104K   13G   1% /run
tmpfs                                            5.0M     0  5.0M   0% /run/lock
/dev/zvol/dakara/incus/custom/default_u1-docker  9.8G   24K  9.3G   1% /var/lib/docker
root@u1:~# grep docker /proc/mounts 
/dev/zvol/dakara/incus/custom/default_u1-docker /var/lib/docker ext4 rw,relatime,discard,stripe=4 0 0
5

Any particular reason to use incus config device ... instead of incus volume attach ...?

6

Nope, it’s just personal preference, they both perform the exact same API call :slight_smile:

7

This seems to confirm everything I’ve read - ZFS! ZFS! ZFS! :slight_smile: For years I eshewed ZFS because it was too new, not kernel-space in Linux, and seemed harder to understand than mdadm + lvm. I note in the video, @stgraber did not benchmark LVM :slight_smile: Is LVM for storage pools passé?

The other question I don’t get is how you can just “create storage pools” out of thin air without specifying the physical device where it’s stored. How does that work? Just build brtfs now. Ok. Go. Create container. See how fast? At one point in the video Stephane created something specific to put it on an nvme partition (went too fast and I didn’t rewind). So clearly, you can specify the storage, but in the examples you don’t have to. Any pointers to docs on this subject would save me from answering further n00b questions.

But it sounds like the guidance is to look LVM in the rearview mirror and embrace ZFS. Guess I’ll have to read up on it in order to manage it properly. I got lost with resilver and zilogs and things.

8

I think I am confused between ZFS as a storage pool for Incus (which requires a block device specified during incus admin init) and ZFS as a general volume manager/filesystem for normal filesystem use (as if I was building a normal fileserver). Can someone point me to documentation / recommendations for storage strategies and mapping Incus storage pools to actual hardware? What if I want some kind of redundancy in my physical storage? Do I use ZFS storage pool on top of RAIDZ linux device? I could probably go back to putting ZFS Incus storage pool on LVM thin pool block devices. But this probably doesn’t make sense.

(Years ago I read LXD Cluster: Redundancy but I don’t think I’m up for multiple machines - just need duplication at the disk layer.)

9

Hi Brendan,

You have some interesting questions. They are hard to answer, though. It really depends on what you are trying to build, and at what scale.

One concept that you might be missing is loop devices.

When you initialize Incus you can specify your first storage pool. There is no right answer for what this should be. If you want to use ZFS, you can, if not that is fine too. Incus can create a ZFS pool for you during initialization, or it can use a ZFS pool that already exists on the system. If you ask Incus to create it for you, you can pass it a disk or a partition that already exists on your system and Incus will do the rest. This is actually a good option and works well with ZFS. If you don’t have a disk or partition to give to Incus, you can still ask Incus to create a ZFS pool. In this case Incus will create a loop device and then create a ZFS pool on this loop device. I think this is the magic you are asking about. It is kind of cool that Incus can do this.

After Incus is initialized, your options grow. You can add other storage pools, and then the process repeats.

On my homelab server I am running Debian 12 with ext4. For Incus, I am using a ZFS loop device. I also have a btrfs storage pool configured like I described earlier. For this machine, I don’t need anything else. If the system dies, I will just rebuild it. No issue.

For my work machines, the requirements are completely different. The on-prem stuff is using Ceph. In the cloud, you don’t have to think as much about these topics. The abstraction level is a bit higher and everything is a VM anyway.

I hope this is helpful. I know I am not answering all of your questions. Maybe others will add their thoughts too.

1 Like
10

I’ve recently read somewhere that ZFS wears disks out rather quickly, within two years. I’m not sure about other file systems, but that would be an argument against using ZFS from my point of view. I tend to use btrfs for everything, but thought of thin-provisioning LVM pools and using them for storage. If someone could please share their experience?

11

Then you read something that was wrong. (It helps if you can provide links to such material, so it can be properly dissected).

Preferably not: the best option is to make a zmirror or raidz/raidz2/raidz3 vdev out of raw disks, or raw partitions, or from LVM volumes on different drives.

With this configuration, ZFS is able to detect and repair silent data corruption, a.k.a. “bitrot”. This is because it has its own checksums of all the data blocks, and can select the correct combination of disks to recreate the correct data.

If you run ZFS on top of a virtual block device which has a hidden RAID underneath, then it will only be able to detect the errors, and not correct them.

(The same applies to a traditional mirror or RAID: it can tell that the mirrors don’t match, or that the parity doesn’t match the data, but it can’t tell you which disk has wrong data)

It’s not a bad option, if you want to mix’n’match zfs and LVM on the same machine. In fact, I do it on my mini home server which has one NVMe drive and one SATA drive:

root@nuc3:~# pvs
  PV             VG  Fmt  Attr PSize    PFree
  /dev/nvme0n1p4 vg0 lvm2 a--  <459.89g 199.88g
  /dev/sda3      vg0 lvm2 a--  <459.66g 199.65g

root@nuc3:~# lvs --segments -o +devices
  LV      VG  Attr       #Str Type   SSize   Devices
  root    vg0 rwi-aor---    2 raid1   20.00g root_rimage_0(0),root_rimage_1(0)
  zpoolm0 vg0 -wi-ao----    1 linear 240.00g /dev/nvme0n1p4(5121)
  zpoolm1 vg0 -wi-ao----    1 linear 240.00g /dev/sda3(5121)

root@nuc3:~# zpool status zfs
  pool: zfs
 state: ONLINE
  scan: scrub repaired 0B in 00:13:20 with 0 errors on Sun Dec 14 00:37:34 2025
config:

	NAME         STATE     READ WRITE CKSUM
	zfs          ONLINE       0     0     0
	  mirror-0   ONLINE       0     0     0
	    zpoolm0  ONLINE       0     0     0
	    zpoolm1  ONLINE       0     0     0

errors: No known data errors

Preferably LVM has raw disks underneath, not mdraid or similar; you can use LVM mirroring or raid on individual logical volumes (including thin pools) to get redundancy over multiple drives - see the “root” logical volume above.

But you can still create non-redundant LVM volumes, which are useful for when you have something like Linstor/DRBD on top, as well as for ZFS.