Is it possible to mount nested ZFS datasets from within a container?

pachulo · August 1, 2024, 12:07am

Hi! I’m setting up a container for nextcloud and I would like to configure multiple ZFS datasets for different data (more context here).

I was expecting that, by setting the zfs.delegate=true on the storage volume, I would be able to create and mount nested ZFS datasets from inside the container, but I can’t get it to work.

Here are the full steps:

On the hotst, I’ve created a new ZFS dataset:
```
sudo zfs create lake/incus
```

Then a new incus storage pool:

incus storage create incus-zfs-dataset zfs source=lake/incus

After that, I’ve created a new volume inside it:

incus storage volume create incus-zfs-dataset nextcloud

And set the zfs.delegate property, so that the container can manage it:

incus storage volume set incus-zfs-dataset custom/nextcloud zfs.delegate=true

I then attached it to the container on the /mnt/nextcloud path:

incus storage volume attach incus-zfs-dataset custom/nextcloud nextcloud-container /mnt/nextcloud

Then, from inside the container:

incus exec nextcloud-container -- su --login ubuntu

I created a nested dataset:

sudo zfs create lake/incus/custom/default_nextcloud/preview

When trying to use the mount command:

sudo mount -F zfs lake/incus/custom/default_nextcloud/preview /mnt/nextcloud/data/appdata_ociemmj53qt4/preview

I got this error:

mount: bad usage
Try 'mount --help' for more information.

And when trying zfs mount:

zfs set mountpoint=/mnt/nextcloud/data/appdata_ociemmj53qt4/preview lake/incus/custom/default_nextcloud/preview

I’m getting the following permission denied error:

cannot mount 'lake/incus/custom/default_nextcloud/preview': Permission denied
property may be set but unable to remount filesystem

I guess I should be able to create a new storage volume inside the incus storage pool from the host, and then attach it to the container, but I was curious to know if I could do it from inside the container.

Thanks!

stgraber · August 1, 2024, 6:14am

That should be mount -t zfs lake/incus/custom/default_nextcloud/preview /mnt/nextcloud/data/appdata_ociemmj53qt4/preview

pachulo · August 1, 2024, 9:17am

Sorry, I forgot to mention that I also tried that:

$ sudo mount -t zfs lake/incus/custom/default_nextcloud/preview /mnt/nextcloud/data/appdata_ociemmj53qt4/preview
filesystem 'lake/incus/custom/default_nextcloud/preview' can not be mounted: Permission denied

stgraber · August 2, 2024, 1:38am

stgraber@dakara:~$ incus launch images:ubuntu/24.04 u1
Launching u1
stgraber@dakara:~$ incus storage volume create default u1-extra size=10GiB zfs.delegate=true
Storage volume u1-extra created
stgraber@dakara:~$ incus config device add u1 extra disk pool=default source=u1-extra path=/mnt
Device extra added to u1

stgraber@dakara:~$ incus exec u1 bash
root@u1:~# apt install zfsutils-linux
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libbrotli1 libcurl4t64 libldap-common libldap2 libnghttp2-14 libnvpair3linux libpsl5t64 librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libssh-4
  libuutil3linux libzfs4linux libzpool5linux publicsuffix zfs-zed
Suggested packages:
  libsasl2-modules-gssapi-mit | libsasl2-modules-gssapi-heimdal libsasl2-modules-ldap libsasl2-modules-otp libsasl2-modules-sql nfs-kernel-server
  samba-common-bin zfs-initramfs | zfs-dracut
The following NEW packages will be installed:
  libbrotli1 libcurl4t64 libldap-common libldap2 libnghttp2-14 libnvpair3linux libpsl5t64 librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libssh-4
  libuutil3linux libzfs4linux libzpool5linux publicsuffix zfs-zed zfsutils-linux
0 upgraded, 18 newly installed, 0 to remove and 0 not upgraded.
Need to get 3900 kB of archives.
After this operation, 11.7 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://archive.ubuntu.com/ubuntu noble/main amd64 libnvpair3linux amd64 2.2.2-0ubuntu9 [61.5 kB]
Get:2 http://archive.ubuntu.com/ubuntu noble/main amd64 libuutil3linux amd64 2.2.2-0ubuntu9 [52.6 kB]
Get:3 http://archive.ubuntu.com/ubuntu noble/main amd64 libzfs4linux amd64 2.2.2-0ubuntu9 [225 kB]
Get:4 http://archive.ubuntu.com/ubuntu noble/main amd64 libzpool5linux amd64 2.2.2-0ubuntu9 [1397 kB]
Get:5 http://archive.ubuntu.com/ubuntu noble/main amd64 zfsutils-linux amd64 2.2.2-0ubuntu9 [551 kB]
Get:6 http://archive.ubuntu.com/ubuntu noble/main amd64 zfs-zed amd64 2.2.2-0ubuntu9 [67.9 kB]
Get:7 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libnghttp2-14 amd64 1.59.0-1ubuntu0.1 [74.3 kB]
Get:8 http://archive.ubuntu.com/ubuntu noble/main amd64 libpsl5t64 amd64 0.21.2-1.1build1 [57.1 kB]
Get:9 http://archive.ubuntu.com/ubuntu noble/main amd64 publicsuffix all 20231001.0357-0.1 [129 kB]
Get:10 http://archive.ubuntu.com/ubuntu noble/main amd64 libbrotli1 amd64 1.1.0-2build2 [331 kB]
Get:11 http://archive.ubuntu.com/ubuntu noble/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg1-5ubuntu3 [20.3 kB]
Get:12 http://archive.ubuntu.com/ubuntu noble/main amd64 libsasl2-2 amd64 2.1.28+dfsg1-5ubuntu3 [53.2 kB]
Get:13 http://archive.ubuntu.com/ubuntu noble/main amd64 libldap2 amd64 2.6.7+dfsg-1~exp1ubuntu8 [195 kB]
Get:14 http://archive.ubuntu.com/ubuntu noble/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2build7 [56.3 kB]
Get:15 http://archive.ubuntu.com/ubuntu noble/main amd64 libssh-4 amd64 0.10.6-2build2 [188 kB]
Get:16 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libcurl4t64 amd64 8.5.0-2ubuntu10.1 [341 kB]
Get:17 http://archive.ubuntu.com/ubuntu noble/main amd64 libldap-common all 2.6.7+dfsg-1~exp1ubuntu8 [31.4 kB]
Get:18 http://archive.ubuntu.com/ubuntu noble/main amd64 libsasl2-modules amd64 2.1.28+dfsg1-5ubuntu3 [69.7 kB]
Fetched 3900 kB in 1s (4713 kB/s)       
Selecting previously unselected package libnvpair3linux.
(Reading database ... 16107 files and directories currently installed.)
Preparing to unpack .../00-libnvpair3linux_2.2.2-0ubuntu9_amd64.deb ...
Unpacking libnvpair3linux (2.2.2-0ubuntu9) ...
Selecting previously unselected package libuutil3linux.
Preparing to unpack .../01-libuutil3linux_2.2.2-0ubuntu9_amd64.deb ...
Unpacking libuutil3linux (2.2.2-0ubuntu9) ...
Selecting previously unselected package libzfs4linux.
Preparing to unpack .../02-libzfs4linux_2.2.2-0ubuntu9_amd64.deb ...
Unpacking libzfs4linux (2.2.2-0ubuntu9) ...
Selecting previously unselected package libzpool5linux.
Preparing to unpack .../03-libzpool5linux_2.2.2-0ubuntu9_amd64.deb ...
Unpacking libzpool5linux (2.2.2-0ubuntu9) ...
Selecting previously unselected package zfsutils-linux.
Preparing to unpack .../04-zfsutils-linux_2.2.2-0ubuntu9_amd64.deb ...
Unpacking zfsutils-linux (2.2.2-0ubuntu9) ...
Selecting previously unselected package zfs-zed.
Preparing to unpack .../05-zfs-zed_2.2.2-0ubuntu9_amd64.deb ...
Unpacking zfs-zed (2.2.2-0ubuntu9) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../06-libnghttp2-14_1.59.0-1ubuntu0.1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.59.0-1ubuntu0.1) ...
Selecting previously unselected package libpsl5t64:amd64.
Preparing to unpack .../07-libpsl5t64_0.21.2-1.1build1_amd64.deb ...
Unpacking libpsl5t64:amd64 (0.21.2-1.1build1) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../08-publicsuffix_20231001.0357-0.1_all.deb ...
Unpacking publicsuffix (20231001.0357-0.1) ...
Selecting previously unselected package libbrotli1:amd64.
Preparing to unpack .../09-libbrotli1_1.1.0-2build2_amd64.deb ...
Unpacking libbrotli1:amd64 (1.1.0-2build2) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../10-libsasl2-modules-db_2.1.28+dfsg1-5ubuntu3_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.28+dfsg1-5ubuntu3) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../11-libsasl2-2_2.1.28+dfsg1-5ubuntu3_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.28+dfsg1-5ubuntu3) ...
Selecting previously unselected package libldap2:amd64.
Preparing to unpack .../12-libldap2_2.6.7+dfsg-1~exp1ubuntu8_amd64.deb ...
Unpacking libldap2:amd64 (2.6.7+dfsg-1~exp1ubuntu8) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../13-librtmp1_2.4+20151223.gitfa8646d.1-2build7_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2build7) ...
Selecting previously unselected package libssh-4:amd64.
Preparing to unpack .../14-libssh-4_0.10.6-2build2_amd64.deb ...
Unpacking libssh-4:amd64 (0.10.6-2build2) ...
Selecting previously unselected package libcurl4t64:amd64.
Preparing to unpack .../15-libcurl4t64_8.5.0-2ubuntu10.1_amd64.deb ...
Unpacking libcurl4t64:amd64 (8.5.0-2ubuntu10.1) ...
Selecting previously unselected package libldap-common.
Preparing to unpack .../16-libldap-common_2.6.7+dfsg-1~exp1ubuntu8_all.deb ...
Unpacking libldap-common (2.6.7+dfsg-1~exp1ubuntu8) ...
Selecting previously unselected package libsasl2-modules:amd64.
Preparing to unpack .../17-libsasl2-modules_2.1.28+dfsg1-5ubuntu3_amd64.deb ...
Unpacking libsasl2-modules:amd64 (2.1.28+dfsg1-5ubuntu3) ...
Setting up libnvpair3linux (2.2.2-0ubuntu9) ...
Setting up libbrotli1:amd64 (1.1.0-2build2) ...
Setting up libsasl2-modules:amd64 (2.1.28+dfsg1-5ubuntu3) ...
Setting up libpsl5t64:amd64 (0.21.2-1.1build1) ...
Setting up libnghttp2-14:amd64 (1.59.0-1ubuntu0.1) ...
Setting up libldap-common (2.6.7+dfsg-1~exp1ubuntu8) ...
Setting up libsasl2-modules-db:amd64 (2.1.28+dfsg1-5ubuntu3) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2build7) ...
Setting up libsasl2-2:amd64 (2.1.28+dfsg1-5ubuntu3) ...
Setting up libssh-4:amd64 (0.10.6-2build2) ...
Setting up libuutil3linux (2.2.2-0ubuntu9) ...
Setting up publicsuffix (20231001.0357-0.1) ...
Setting up libldap2:amd64 (2.6.7+dfsg-1~exp1ubuntu8) ...
Setting up libzpool5linux (2.2.2-0ubuntu9) ...
Setting up libcurl4t64:amd64 (8.5.0-2ubuntu10.1) ...
Setting up libzfs4linux (2.2.2-0ubuntu9) ...
Setting up zfsutils-linux (2.2.2-0ubuntu9) ...
modprobe: FATAL: Module zfs not found in directory /lib/modules/6.9.12-zabbly+
Created symlink /etc/systemd/system/zfs-import.target.wants/zfs-import-cache.service → /usr/lib/systemd/system/zfs-import-cache.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-import.target → /usr/lib/systemd/system/zfs-import.target.
Created symlink /etc/systemd/system/zfs-mount.service.wants/zfs-load-module.service → /usr/lib/systemd/system/zfs-load-module.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-load-module.service → /usr/lib/systemd/system/zfs-load-module.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-mount.service → /usr/lib/systemd/system/zfs-mount.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-share.service → /usr/lib/systemd/system/zfs-share.service.
Created symlink /etc/systemd/system/zfs-volumes.target.wants/zfs-volume-wait.service → /usr/lib/systemd/system/zfs-volume-wait.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-volumes.target → /usr/lib/systemd/system/zfs-volumes.target.
Created symlink /etc/systemd/system/multi-user.target.wants/zfs.target → /usr/lib/systemd/system/zfs.target.
zfs-import-scan.service is a disabled or a static unit, not starting it.
Setting up zfs-zed (2.2.2-0ubuntu9) ...
Created symlink /etc/systemd/system/zed.service → /usr/lib/systemd/system/zfs-zed.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-zed.service → /usr/lib/systemd/system/zfs-zed.service.
Processing triggers for libc-bin (2.39-0ubuntu8.2) ...

root@u1:~# zfs list -t all
NAME                                   USED  AVAIL  REFER  MOUNTPOINT
dakara                                 273G   533G    98K  none
dakara/incus                           119G   533G   105K  legacy
dakara/incus/custom                   7.83G   533G   105K  legacy
dakara/incus/custom/default_u1-extra    98K  10.0G    98K  legacy
root@u1:~# zfs create dakara/incus/custom/default_u1-extra/foo
root@u1:~# zfs list -t all
NAME                                       USED  AVAIL  REFER  MOUNTPOINT
dakara                                     273G   533G    98K  none
dakara/incus                               119G   533G   105K  legacy
dakara/incus/custom                       7.83G   533G   105K  legacy
dakara/incus/custom/default_u1-extra       196K  10.0G    98K  legacy
dakara/incus/custom/default_u1-extra/foo    98K  10.0G    98K  legacy
root@u1:~# mount -t zfs dakara/incus/custom/default_u1-extra/foo /srv/
root@u1:~# df -h
Filesystem                                Size  Used Avail Use% Mounted on
dakara/incus/containers/u1                 21G  334M   20G   2% /
none                                      492K  4.0K  488K   1% /dev
efivarfs                                  128K   36K   88K  29% /sys/firmware/efi/efivars
tmpfs                                     100K     0  100K   0% /dev/incus
tmpfs                                     100K     0  100K   0% /dev/.incus-mounts
tmpfs                                      32G     0   32G   0% /dev/shm
tmpfs                                      13G  124K   13G   1% /run
tmpfs                                     5.0M     0  5.0M   0% /run/lock
dakara/incus/custom/default_u1-extra       10G  128K   10G   1% /mnt
dakara/incus/custom/default_u1-extra/foo   10G  128K   10G   1% /srv
root@u1:~#

pachulo · August 5, 2024, 10:21am

First of all, thanks a lot for your help @stgraber !

I’ve tried to reproduce your example, but got the same permission denied error:

$ incus launch images:ubuntu/24.04 u1
Launching u1
$ incus storage volume create incus-zfs-dataset u1-extra size=10GiB zfs.delegate=true
Storage volume u1-extra created
$ incus config device add u1 extra disk pool=incus-zfs-dataset source=u1-extra path=/mnt
Device extra added to u1

$ incus exec u1 bash
root@u1:~# apt install zfsutils-linux
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libbrotli1 libcurl4t64 libldap-common libldap2 libnghttp2-14 libnvpair3linux libpsl5t64 librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libssh-4 libuutil3linux libzfs4linux libzpool5linux
  publicsuffix zfs-zed
Suggested packages:
  libsasl2-modules-gssapi-mit | libsasl2-modules-gssapi-heimdal libsasl2-modules-ldap libsasl2-modules-otp libsasl2-modules-sql nfs-kernel-server samba-common-bin zfs-initramfs | zfs-dracut
The following NEW packages will be installed:
  libbrotli1 libcurl4t64 libldap-common libldap2 libnghttp2-14 libnvpair3linux libpsl5t64 librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libssh-4 libuutil3linux libzfs4linux libzpool5linux
  publicsuffix zfs-zed zfsutils-linux
0 upgraded, 18 newly installed, 0 to remove and 0 not upgraded.
Need to get 3900 kB of archives.
After this operation, 11.7 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://archive.ubuntu.com/ubuntu noble/main amd64 libnvpair3linux amd64 2.2.2-0ubuntu9 [61.5 kB]
Get:2 http://archive.ubuntu.com/ubuntu noble/main amd64 libuutil3linux amd64 2.2.2-0ubuntu9 [52.6 kB]
Get:3 http://archive.ubuntu.com/ubuntu noble/main amd64 libzfs4linux amd64 2.2.2-0ubuntu9 [225 kB]
Get:4 http://archive.ubuntu.com/ubuntu noble/main amd64 libzpool5linux amd64 2.2.2-0ubuntu9 [1397 kB]
Get:5 http://archive.ubuntu.com/ubuntu noble/main amd64 zfsutils-linux amd64 2.2.2-0ubuntu9 [551 kB]
Get:6 http://archive.ubuntu.com/ubuntu noble/main amd64 zfs-zed amd64 2.2.2-0ubuntu9 [67.9 kB]
Get:7 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libnghttp2-14 amd64 1.59.0-1ubuntu0.1 [74.3 kB]
Get:8 http://archive.ubuntu.com/ubuntu noble/main amd64 libpsl5t64 amd64 0.21.2-1.1build1 [57.1 kB]
Get:9 http://archive.ubuntu.com/ubuntu noble/main amd64 publicsuffix all 20231001.0357-0.1 [129 kB]
Get:10 http://archive.ubuntu.com/ubuntu noble/main amd64 libbrotli1 amd64 1.1.0-2build2 [331 kB]
Get:11 http://archive.ubuntu.com/ubuntu noble/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg1-5ubuntu3 [20.3 kB]
Get:12 http://archive.ubuntu.com/ubuntu noble/main amd64 libsasl2-2 amd64 2.1.28+dfsg1-5ubuntu3 [53.2 kB]
Get:13 http://archive.ubuntu.com/ubuntu noble/main amd64 libldap2 amd64 2.6.7+dfsg-1~exp1ubuntu8 [195 kB]
Get:14 http://archive.ubuntu.com/ubuntu noble/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2build7 [56.3 kB]
Get:15 http://archive.ubuntu.com/ubuntu noble/main amd64 libssh-4 amd64 0.10.6-2build2 [188 kB]
Get:16 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libcurl4t64 amd64 8.5.0-2ubuntu10.1 [341 kB]
Get:17 http://archive.ubuntu.com/ubuntu noble/main amd64 libldap-common all 2.6.7+dfsg-1~exp1ubuntu8 [31.4 kB]
Get:18 http://archive.ubuntu.com/ubuntu noble/main amd64 libsasl2-modules amd64 2.1.28+dfsg1-5ubuntu3 [69.7 kB]
Fetched 3900 kB in 1s (2874 kB/s)
Selecting previously unselected package libnvpair3linux.
(Reading database ... 16107 files and directories currently installed.)
Preparing to unpack .../00-libnvpair3linux_2.2.2-0ubuntu9_amd64.deb ...
Unpacking libnvpair3linux (2.2.2-0ubuntu9) ...
Selecting previously unselected package libuutil3linux.
Preparing to unpack .../01-libuutil3linux_2.2.2-0ubuntu9_amd64.deb ...
Unpacking libuutil3linux (2.2.2-0ubuntu9) ...
Selecting previously unselected package libzfs4linux.
Preparing to unpack .../02-libzfs4linux_2.2.2-0ubuntu9_amd64.deb ...
Unpacking libzfs4linux (2.2.2-0ubuntu9) ...
Selecting previously unselected package libzpool5linux.
Preparing to unpack .../03-libzpool5linux_2.2.2-0ubuntu9_amd64.deb ...
Unpacking libzpool5linux (2.2.2-0ubuntu9) ...
Selecting previously unselected package zfsutils-linux.
Preparing to unpack .../04-zfsutils-linux_2.2.2-0ubuntu9_amd64.deb ...
Unpacking zfsutils-linux (2.2.2-0ubuntu9) ...
Selecting previously unselected package zfs-zed.
Preparing to unpack .../05-zfs-zed_2.2.2-0ubuntu9_amd64.deb ...
Unpacking zfs-zed (2.2.2-0ubuntu9) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../06-libnghttp2-14_1.59.0-1ubuntu0.1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.59.0-1ubuntu0.1) ...
Selecting previously unselected package libpsl5t64:amd64.
Preparing to unpack .../07-libpsl5t64_0.21.2-1.1build1_amd64.deb ...
Unpacking libpsl5t64:amd64 (0.21.2-1.1build1) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../08-publicsuffix_20231001.0357-0.1_all.deb ...
Unpacking publicsuffix (20231001.0357-0.1) ...
Selecting previously unselected package libbrotli1:amd64.
Preparing to unpack .../09-libbrotli1_1.1.0-2build2_amd64.deb ...
Unpacking libbrotli1:amd64 (1.1.0-2build2) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../10-libsasl2-modules-db_2.1.28+dfsg1-5ubuntu3_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.28+dfsg1-5ubuntu3) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../11-libsasl2-2_2.1.28+dfsg1-5ubuntu3_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.28+dfsg1-5ubuntu3) ...
Selecting previously unselected package libldap2:amd64.
Preparing to unpack .../12-libldap2_2.6.7+dfsg-1~exp1ubuntu8_amd64.deb ...
Unpacking libldap2:amd64 (2.6.7+dfsg-1~exp1ubuntu8) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../13-librtmp1_2.4+20151223.gitfa8646d.1-2build7_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2build7) ...
Selecting previously unselected package libssh-4:amd64.
Preparing to unpack .../14-libssh-4_0.10.6-2build2_amd64.deb ...
Unpacking libssh-4:amd64 (0.10.6-2build2) ...
Selecting previously unselected package libcurl4t64:amd64.
Preparing to unpack .../15-libcurl4t64_8.5.0-2ubuntu10.1_amd64.deb ...
Unpacking libcurl4t64:amd64 (8.5.0-2ubuntu10.1) ...
Selecting previously unselected package libldap-common.
Preparing to unpack .../16-libldap-common_2.6.7+dfsg-1~exp1ubuntu8_all.deb ...
Unpacking libldap-common (2.6.7+dfsg-1~exp1ubuntu8) ...
Selecting previously unselected package libsasl2-modules:amd64.
Preparing to unpack .../17-libsasl2-modules_2.1.28+dfsg1-5ubuntu3_amd64.deb ...
Unpacking libsasl2-modules:amd64 (2.1.28+dfsg1-5ubuntu3) ...
Setting up libnvpair3linux (2.2.2-0ubuntu9) ...
Setting up libbrotli1:amd64 (1.1.0-2build2) ...
Setting up libsasl2-modules:amd64 (2.1.28+dfsg1-5ubuntu3) ...
Setting up libpsl5t64:amd64 (0.21.2-1.1build1) ...
Setting up libnghttp2-14:amd64 (1.59.0-1ubuntu0.1) ...
Setting up libldap-common (2.6.7+dfsg-1~exp1ubuntu8) ...
Setting up libsasl2-modules-db:amd64 (2.1.28+dfsg1-5ubuntu3) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2build7) ...
Setting up libsasl2-2:amd64 (2.1.28+dfsg1-5ubuntu3) ...
Setting up libssh-4:amd64 (0.10.6-2build2) ...
Setting up libuutil3linux (2.2.2-0ubuntu9) ...
Setting up publicsuffix (20231001.0357-0.1) ...
Setting up libldap2:amd64 (2.6.7+dfsg-1~exp1ubuntu8) ...
Setting up libzpool5linux (2.2.2-0ubuntu9) ...
Setting up libcurl4t64:amd64 (8.5.0-2ubuntu10.1) ...
Setting up libzfs4linux (2.2.2-0ubuntu9) ...
Setting up zfsutils-linux (2.2.2-0ubuntu9) ...
modprobe: FATAL: Module zfs not found in directory /lib/modules/6.8.0-39-generic
Created symlink /etc/systemd/system/zfs-import.target.wants/zfs-import-cache.service → /usr/lib/systemd/system/zfs-import-cache.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-import.target → /usr/lib/systemd/system/zfs-import.target.
Created symlink /etc/systemd/system/zfs-mount.service.wants/zfs-load-module.service → /usr/lib/systemd/system/zfs-load-module.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-load-module.service → /usr/lib/systemd/system/zfs-load-module.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-mount.service → /usr/lib/systemd/system/zfs-mount.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-share.service → /usr/lib/systemd/system/zfs-share.service.
Created symlink /etc/systemd/system/zfs-volumes.target.wants/zfs-volume-wait.service → /usr/lib/systemd/system/zfs-volume-wait.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-volumes.target → /usr/lib/systemd/system/zfs-volumes.target.
Created symlink /etc/systemd/system/multi-user.target.wants/zfs.target → /usr/lib/systemd/system/zfs.target.
zfs-import-scan.service is a disabled or a static unit, not starting it.
Setting up zfs-zed (2.2.2-0ubuntu9) ...
Created symlink /etc/systemd/system/zed.service → /usr/lib/systemd/system/zfs-zed.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-zed.service → /usr/lib/systemd/system/zfs-zed.service.
Processing triggers for libc-bin (2.39-0ubuntu8.2) ...
root@u1:~# zfs list -t all
NAME                                 USED  AVAIL  REFER  MOUNTPOINT
lake                                64.4G  14.3T   234K  /lake
lake/incus                          64.4G  14.3T   234K  legacy
lake/incus/custom                   64.4G  14.3T   234K  legacy
lake/incus/custom/default_u1-extra   234K  10.0G   234K  legacy
root@u1:~# zfs create lake/incus/custom/default_u1-extra/foo
root@u1:~# zfs list -t all
NAME                                     USED  AVAIL  REFER  MOUNTPOINT
lake                                    64.4G  14.3T   234K  /lake
lake/incus                              64.4G  14.3T   234K  legacy
lake/incus/custom                       64.4G  14.3T   234K  legacy
lake/incus/custom/default_u1-extra       469K  10.0G   234K  legacy
lake/incus/custom/default_u1-extra/foo   234K  10.0G   234K  legacy
root@u1:~# mount -t zfs lake/incus/custom/default_u1-extra/foo /srv/
filesystem 'lake/incus/custom/default_u1-extra/foo' can not be mounted: Permission denied

And the only difference I can think of is that I’m not working with the default storage pool, but an additional one I created:

$ sudo zfs create lake/incus
$ incus storage create incus-zfs-dataset zfs source=lake/incus
$ incus storage list
+-------------------+--------+-------------+---------+---------+
|       NAME        | DRIVER | DESCRIPTION | USED BY |  STATE  |
+-------------------+--------+-------------+---------+---------+
| default           | lvm    |             | 5       | CREATED |
+-------------------+--------+-------------+---------+---------+
| incus-zfs-dataset | zfs    |             | 2       | CREATED |
+-------------------+--------+-------------+---------+---------+
$ incus storage show incus-zfs-dataset
config:
  source: lake/incus
  volatile.initial_source: lake/incus
  zfs.pool_name: lake/incus
description: ""
name: incus-zfs-dataset
driver: zfs
used_by:
- /1.0/storage-pools/incus-zfs-dataset/volumes/custom/u1-extra
status: Created
locations:
- none

stgraber · August 5, 2024, 4:01pm

Can you look for any DENIED entries in dmesg?

pachulo · August 5, 2024, 8:09pm

Yes indeed:

[326838.334609] audit: type=1400 audit(1722503823.764:560): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="incus-nextcloud-container_</var/lib/incus>" name="/mnt/nextcloud/data/appdata_ociemmj53qt4/preview/" pid=123975 comm="mount.zfs" fstype="zfs" srcname="lake/incus/custom/default_nextcloud/preview"
[632638.920310] audit: type=1400 audit(1722809624.000:564): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="incus-nextcloud-container_</var/lib/incus>" name="/srv/" pid=163900 comm="mount.zfs" fstype="zfs" srcname="lake/incus/custom/default_nextcloud/preview"
[675384.710007] audit: type=1400 audit(1722852369.743:577): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="incus-u1_</var/lib/incus>" name="/srv/" pid=182898 comm="mount.zfs" fstype="zfs" srcname="lake/incus/custom/default_u1-extra/foo"

By the way, the host runs Ubuntu 24.04 with the 6.8.0-39-generic kernel.

stgraber · August 5, 2024, 9:17pm

Okay, try putting raw.apparmor=mount fstype=zfs, in your container config, that should take care of that.

I’ll look at adding that to the generated profile, not sure why it’s not there already and even less sure why it’s not needed on my system…

stgraber · August 5, 2024, 9:32pm

pachulo · August 6, 2024, 9:07am

I added it like this:

$ incus config set u1 raw.apparmor "mount fstype=zfs,"

And now it works!

I understand then that because this is now merged, the workaround won’t be necessary on next Incus version, right?

Thanks for all the support!

stgraber · August 6, 2024, 1:30pm

That’s right, the fix will be in Incus 6.4.