Recommended general storage pool

Incus
1

In my new Incus build, I wish to be able to run Docker containers inside Incus containers. In @stgraber 's excellent video on the subject, he uses brtfs as Docker doesn’t presently cooperate with ZFS. In my existing 5yo LXD install, I used LVM with thin provisioning as my storage pool. I haven’t really been able to find guidance on recommendations for general-purpose storage pools. I think I originally picked LVM/thinpool for ease of snapshotting for backups and checkpoints if I was going to do a software upgrade in a container that might go sour. It’s been great for this, but maybe brtfs is better?

  1. Use brtfs for everything - docker will work fine like in the video
  2. Use LVM for everything - docker will still work fine(?)
  3. Use LVM for most things, but create separate brtfs pool for docker-using containers

Edit: Bonus points for simplicity!

TIA,
-brmiller

2

I use ZFS for everything except podman/docker workloads.

I would recommend starting over with Incus and ZFS, then follow the directions in the video. There is another video somewhere about different storage options. ZFS seemed to win over LVM.

incus launch images:debian/12/cloud d1
incus storage create podman btrf
incus storage volume create podman tmp-btfs-volume
incus storage volume attach podman tmp-btfs-volume d1 /var/lib/containers
incus config set d1 security.nesting=true security.syscalls.intercept.mknod=true security.syscalls.intercept.setxattr=true
incus restart d1
incus exec d1 -- su -l debian

Then from inside your container:

sudo apt update
sudo apt install podman
podman run -it docker.io/ubuntu /bin/bash

All seems to work. I did not experience any performance issues. I have in the past before I learned about the special needs of these overlay filesystems.

1 Like
3

I think this is the video I was remembering.

4

Worth noting that you can now use a block-mode ZFS volume that will get you an ext4 volume from your ZFS pool.

stgraber@dakara:~$ incus launch images:ubuntu/22.04 u1
Launching u1
stgraber@dakara:~$ incus storage volume create default u1-docker zfs.block_mode=true size=10GiB
Storage volume u1-docker created
stgraber@dakara:~$ incus config device add u1 docker disk pool=default source=u1-docker path=/var/lib/docker
Device docker added to u1
stgraber@dakara:~$ incus exec u1 bash
root@u1:~# df -h
Filesystem                                       Size  Used Avail Use% Mounted on
dakara/incus/containers/u1                       670G  264M  670G   1% /
none                                             492K  4.0K  488K   1% /dev
efivarfs                                         128K   33K   91K  27% /sys/firmware/efi/efivars
tmpfs                                            100K     0  100K   0% /dev/incus
tmpfs                                            100K     0  100K   0% /dev/.incus-mounts
tmpfs                                             32G     0   32G   0% /dev/shm
tmpfs                                             13G  104K   13G   1% /run
tmpfs                                            5.0M     0  5.0M   0% /run/lock
/dev/zvol/dakara/incus/custom/default_u1-docker  9.8G   24K  9.3G   1% /var/lib/docker
root@u1:~# grep docker /proc/mounts 
/dev/zvol/dakara/incus/custom/default_u1-docker /var/lib/docker ext4 rw,relatime,discard,stripe=4 0 0
5

Any particular reason to use incus config device ... instead of incus volume attach ...?

6

Nope, it’s just personal preference, they both perform the exact same API call :slight_smile:

7

This seems to confirm everything I’ve read - ZFS! ZFS! ZFS! :slight_smile: For years I eshewed ZFS because it was too new, not kernel-space in Linux, and seemed harder to understand than mdadm + lvm. I note in the video, @stgraber did not benchmark LVM :slight_smile: Is LVM for storage pools passé?

The other question I don’t get is how you can just “create storage pools” out of thin air without specifying the physical device where it’s stored. How does that work? Just build brtfs now. Ok. Go. Create container. See how fast? At one point in the video Stephane created something specific to put it on an nvme partition (went too fast and I didn’t rewind). So clearly, you can specify the storage, but in the examples you don’t have to. Any pointers to docs on this subject would save me from answering further n00b questions.

But it sounds like the guidance is to look LVM in the rearview mirror and embrace ZFS. Guess I’ll have to read up on it in order to manage it properly. I got lost with resilver and zilogs and things.

8

I think I am confused between ZFS as a storage pool for Incus (which requires a block device specified during incus admin init) and ZFS as a general volume manager/filesystem for normal filesystem use (as if I was building a normal fileserver). Can someone point me to documentation / recommendations for storage strategies and mapping Incus storage pools to actual hardware? What if I want some kind of redundancy in my physical storage? Do I use ZFS storage pool on top of RAIDZ linux device? I could probably go back to putting ZFS Incus storage pool on LVM thin pool block devices. But this probably doesn’t make sense.

(Years ago I read LXD Cluster: Redundancy but I don’t think I’m up for multiple machines - just need duplication at the disk layer.)

9

Hi Brendan,

You have some interesting questions. They are hard to answer, though. It really depends on what you are trying to build, and at what scale.

One concept that you might be missing is loop devices.

When you initialize Incus you can specify your first storage pool. There is no right answer for what this should be. If you want to use ZFS, you can, if not that is fine too. Incus can create a ZFS pool for you during initialization, or it can use a ZFS pool that already exists on the system. If you ask Incus to create it for you, you can pass it a disk or a partition that already exists on your system and Incus will do the rest. This is actually a good option and works well with ZFS. If you don’t have a disk or partition to give to Incus, you can still ask Incus to create a ZFS pool. In this case Incus will create a loop device and then create a ZFS pool on this loop device. I think this is the magic you are asking about. It is kind of cool that Incus can do this.

After Incus is initialized, your options grow. You can add other storage pools, and then the process repeats.

On my homelab server I am running Debian 12 with ext4. For Incus, I am using a ZFS loop device. I also have a btrfs storage pool configured like I described earlier. For this machine, I don’t need anything else. If the system dies, I will just rebuild it. No issue.

For my work machines, the requirements are completely different. The on-prem stuff is using Ceph. In the cloud, you don’t have to think as much about these topics. The abstraction level is a bit higher and everything is a VM anyway.

I hope this is helpful. I know I am not answering all of your questions. Maybe others will add their thoughts too.

1 Like