Nested ZFS does not work on Alpine Linux 3.20

I am trying to reproduce this: Is it possible to mount nested ZFS datasets from within a container? - #4 by stgraber

My host is Apline Linux 3.20 with the 6.6.57-0-lts Kernel and ZFS zfs-2.2.5-1.
Edit: Incus 6.0.2

I get at the end in the container running zfs list:

Permission denied the ZFS utilities must be run as root.

Is this because Alpine’s ZFS is further ahead and Ubuntu’s 2.2.2 is not compatible?

cs0001:~# incus launch images:ubuntu/24.04 u1
Launching u1
cs0001:~# incus config device add u1 extra disk pool=base-pool source=u1-extra path=/mnt
Device extra added to u1
cs0001:~# incus exec u1 bash
root@u1:~# apt install zfsutils-linux
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libbrotli1 libcurl4t64 libldap-common libldap2 libnghttp2-14 libnvpair3linux libpsl5t64 librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libssh-4 libuutil3linux libzfs4linux libzpool5linux
  publicsuffix zfs-zed
Suggested packages:
  libsasl2-modules-gssapi-mit | libsasl2-modules-gssapi-heimdal libsasl2-modules-ldap libsasl2-modules-otp libsasl2-modules-sql nfs-kernel-server samba-common-bin zfs-initramfs | zfs-dracut
The following NEW packages will be installed:
  libbrotli1 libcurl4t64 libldap-common libldap2 libnghttp2-14 libnvpair3linux libpsl5t64 librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libssh-4 libuutil3linux libzfs4linux libzpool5linux
  publicsuffix zfs-zed zfsutils-linux
0 upgraded, 18 newly installed, 0 to remove and 0 not upgraded.
Need to get 3903 kB of archives.
After this operation, 11.7 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libnvpair3linux amd64 2.2.2-0ubuntu9.1 [61.6 kB]
Get:2 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libuutil3linux amd64 2.2.2-0ubuntu9.1 [52.7 kB]
Get:3 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libzfs4linux amd64 2.2.2-0ubuntu9.1 [226 kB]
Get:4 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libzpool5linux amd64 2.2.2-0ubuntu9.1 [1397 kB]
Get:5 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 zfsutils-linux amd64 2.2.2-0ubuntu9.1 [551 kB]
Get:6 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 zfs-zed amd64 2.2.2-0ubuntu9.1 [67.9 kB]
Get:7 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libnghttp2-14 amd64 1.59.0-1ubuntu0.1 [74.3 kB]
Get:8 http://archive.ubuntu.com/ubuntu noble/main amd64 libpsl5t64 amd64 0.21.2-1.1build1 [57.1 kB]
Get:9 http://archive.ubuntu.com/ubuntu noble/main amd64 publicsuffix all 20231001.0357-0.1 [129 kB]
Get:10 http://archive.ubuntu.com/ubuntu noble/main amd64 libbrotli1 amd64 1.1.0-2build2 [331 kB]
Get:11 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg1-5ubuntu3.1 [20.4 kB]
Get:12 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libsasl2-2 amd64 2.1.28+dfsg1-5ubuntu3.1 [53.2 kB]
Get:13 http://archive.ubuntu.com/ubuntu noble/main amd64 libldap2 amd64 2.6.7+dfsg-1~exp1ubuntu8 [195 kB]
Get:14 http://archive.ubuntu.com/ubuntu noble/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2build7 [56.3 kB]
Get:15 http://archive.ubuntu.com/ubuntu noble/main amd64 libssh-4 amd64 0.10.6-2build2 [188 kB]
Get:16 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libcurl4t64 amd64 8.5.0-2ubuntu10.4 [341 kB]
Get:17 http://archive.ubuntu.com/ubuntu noble/main amd64 libldap-common all 2.6.7+dfsg-1~exp1ubuntu8 [31.4 kB]
Get:18 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libsasl2-modules amd64 2.1.28+dfsg1-5ubuntu3.1 [69.9 kB]
Fetched 3903 kB in 0s (12.2 MB/s)            
Selecting previously unselected package libnvpair3linux.
(Reading database ... 16188 files and directories currently installed.)
Preparing to unpack .../00-libnvpair3linux_2.2.2-0ubuntu9.1_amd64.deb ...
Unpacking libnvpair3linux (2.2.2-0ubuntu9.1) ...
Selecting previously unselected package libuutil3linux.
Preparing to unpack .../01-libuutil3linux_2.2.2-0ubuntu9.1_amd64.deb ...
Unpacking libuutil3linux (2.2.2-0ubuntu9.1) ...
Selecting previously unselected package libzfs4linux.
Preparing to unpack .../02-libzfs4linux_2.2.2-0ubuntu9.1_amd64.deb ...
Unpacking libzfs4linux (2.2.2-0ubuntu9.1) ...
Selecting previously unselected package libzpool5linux.
Preparing to unpack .../03-libzpool5linux_2.2.2-0ubuntu9.1_amd64.deb ...
Unpacking libzpool5linux (2.2.2-0ubuntu9.1) ...
Selecting previously unselected package zfsutils-linux.
Preparing to unpack .../04-zfsutils-linux_2.2.2-0ubuntu9.1_amd64.deb ...
Unpacking zfsutils-linux (2.2.2-0ubuntu9.1) ...
Selecting previously unselected package zfs-zed.
Preparing to unpack .../05-zfs-zed_2.2.2-0ubuntu9.1_amd64.deb ...
Unpacking zfs-zed (2.2.2-0ubuntu9.1) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../06-libnghttp2-14_1.59.0-1ubuntu0.1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.59.0-1ubuntu0.1) ...
Selecting previously unselected package libpsl5t64:amd64.
Preparing to unpack .../07-libpsl5t64_0.21.2-1.1build1_amd64.deb ...
Unpacking libpsl5t64:amd64 (0.21.2-1.1build1) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../08-publicsuffix_20231001.0357-0.1_all.deb ...
Unpacking publicsuffix (20231001.0357-0.1) ...
Selecting previously unselected package libbrotli1:amd64.
Preparing to unpack .../09-libbrotli1_1.1.0-2build2_amd64.deb ...
Unpacking libbrotli1:amd64 (1.1.0-2build2) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../10-libsasl2-modules-db_2.1.28+dfsg1-5ubuntu3.1_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.28+dfsg1-5ubuntu3.1) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../11-libsasl2-2_2.1.28+dfsg1-5ubuntu3.1_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.28+dfsg1-5ubuntu3.1) ...
Selecting previously unselected package libldap2:amd64.
Preparing to unpack .../12-libldap2_2.6.7+dfsg-1~exp1ubuntu8_amd64.deb ...
Unpacking libldap2:amd64 (2.6.7+dfsg-1~exp1ubuntu8) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../13-librtmp1_2.4+20151223.gitfa8646d.1-2build7_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2build7) ...
Selecting previously unselected package libssh-4:amd64.
Preparing to unpack .../14-libssh-4_0.10.6-2build2_amd64.deb ...
Unpacking libssh-4:amd64 (0.10.6-2build2) ...
Selecting previously unselected package libcurl4t64:amd64.
Preparing to unpack .../15-libcurl4t64_8.5.0-2ubuntu10.4_amd64.deb ...
Unpacking libcurl4t64:amd64 (8.5.0-2ubuntu10.4) ...
Selecting previously unselected package libldap-common.
Preparing to unpack .../16-libldap-common_2.6.7+dfsg-1~exp1ubuntu8_all.deb ...
Unpacking libldap-common (2.6.7+dfsg-1~exp1ubuntu8) ...
Selecting previously unselected package libsasl2-modules:amd64.
Preparing to unpack .../17-libsasl2-modules_2.1.28+dfsg1-5ubuntu3.1_amd64.deb ...
Unpacking libsasl2-modules:amd64 (2.1.28+dfsg1-5ubuntu3.1) ...
Setting up libnvpair3linux (2.2.2-0ubuntu9.1) ...
Setting up libbrotli1:amd64 (1.1.0-2build2) ...
Setting up libsasl2-modules:amd64 (2.1.28+dfsg1-5ubuntu3.1) ...
Setting up libpsl5t64:amd64 (0.21.2-1.1build1) ...
Setting up libnghttp2-14:amd64 (1.59.0-1ubuntu0.1) ...
Setting up libldap-common (2.6.7+dfsg-1~exp1ubuntu8) ...
Setting up libsasl2-modules-db:amd64 (2.1.28+dfsg1-5ubuntu3.1) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2build7) ...
Setting up libsasl2-2:amd64 (2.1.28+dfsg1-5ubuntu3.1) ...
Setting up libssh-4:amd64 (0.10.6-2build2) ...
Setting up libuutil3linux (2.2.2-0ubuntu9.1) ...
Setting up publicsuffix (20231001.0357-0.1) ...
Setting up libldap2:amd64 (2.6.7+dfsg-1~exp1ubuntu8) ...
Setting up libzpool5linux (2.2.2-0ubuntu9.1) ...
Setting up libcurl4t64:amd64 (8.5.0-2ubuntu10.4) ...
Setting up libzfs4linux (2.2.2-0ubuntu9.1) ...
Setting up zfsutils-linux (2.2.2-0ubuntu9.1) ...
modprobe: FATAL: Module zfs not found in directory /lib/modules/6.6.57-0-lts
Created symlink /etc/systemd/system/zfs-import.target.wants/zfs-import-cache.service → /usr/lib/systemd/system/zfs-import-cache.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-import.target → /usr/lib/systemd/system/zfs-import.target.
Created symlink /etc/systemd/system/zfs-mount.service.wants/zfs-load-module.service → /usr/lib/systemd/system/zfs-load-module.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-load-module.service → /usr/lib/systemd/system/zfs-load-module.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-mount.service → /usr/lib/systemd/system/zfs-mount.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-share.service → /usr/lib/systemd/system/zfs-share.service.
Created symlink /etc/systemd/system/zfs-volumes.target.wants/zfs-volume-wait.service → /usr/lib/systemd/system/zfs-volume-wait.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-volumes.target → /usr/lib/systemd/system/zfs-volumes.target.
Created symlink /etc/systemd/system/multi-user.target.wants/zfs.target → /usr/lib/systemd/system/zfs.target.
zfs-import-scan.service is a disabled or a static unit, not starting it.
Could not execute systemctl:  at /usr/bin/deb-systemd-invoke line 148.
Setting up zfs-zed (2.2.2-0ubuntu9.1) ...
Created symlink /etc/systemd/system/zed.service → /usr/lib/systemd/system/zfs-zed.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-zed.service → /usr/lib/systemd/system/zfs-zed.service.
Processing triggers for libc-bin (2.39-0ubuntu8.3) ...
root@u1:~# zfs list
Permission denied the ZFS utilities must be run as root.

Probably not the different versions, as it is the same story with Alpine:

cs0001:~# incus launch images:alpine/3.20 alp1
Launching alp1
cs0001:~# incus storage volume create base-pool alp1-extra size=10GiB zfs.delegate=true
Storage volume alp1-extra created
cs0001:~# incus config device add alp1 extra disk pool=base-pool source=alp1-extra path=/mnt
Device extra added to alp1
cs0001:~# incus exec alp1 ash
~ # apk add zfs
fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/community/x86_64/APKINDEX.tar.gz
(1/14) Installing libeconf (0.6.3-r0)
(2/14) Installing libblkid (2.40.1-r1)
(3/14) Installing libintl (0.22.5-r0)
(4/14) Installing libtirpc-conf (1.3.4-r0)
(5/14) Installing krb5-conf (1.0-r2)
(6/14) Installing libcom_err (1.47.0-r5)
(7/14) Installing keyutils-libs (1.6.3-r3)
(8/14) Installing libverto (0.3.2-r2)
(9/14) Installing krb5-libs (1.21.3-r0)
(10/14) Installing libtirpc (1.3.4-r0)
(11/14) Installing libuuid (2.40.1-r1)
(12/14) Installing zfs-libs (2.2.5-r0)
(13/14) Installing zfs (2.2.5-r0)
(14/14) Installing zfs-openrc (2.2.5-r0)
Executing busybox-1.36.1-r29.trigger
OK: 18 MiB in 43 packages
~ # zfs list
Permission denied the ZFS utilities must be run as root.

What version of Incus are you using? Did you try the apparmor setting in the other thread?

I am using the Apline Linux supplied version of Incus: 6.0.2
I didn’t think it necessary to do the AppArmor stuff, as it is not installed on the host, dmesg does not have any output with “DENIED”. I did it anyways now, but it did not work.

Is there any way for you to test a more current version of Incus?

I think there are a few other Alpine users in the forums.

I am not sure what zfs delegation is. Maybe that is the missing piece. I bet @stgraber knows what is going on. Sorry I could not help more.

1 Like

I think zfs.delegation was supported since 0.something already.
And no worries, thanks for your effort! :heart:

1 Like

Do you have any ideas in which direction to look @stgraber ?
Thanks in advance!

Edit: Limits are not being applied to containers, there must be something wrong with my Alpine Setup.

Did my host system with Ubuntu 24.04.1 and Incus 6.0.2 from zabbly. Works perfectly in an Ubuntu 24.04 container. Just wished it would work with other setups as well.