ZFS attached storage volume is not mounted on host

Hi there,

I’ve a problem since upgrading to lxd 3.14/snap version.

On lxd 3.8 I can do this:

lxc --version

3.8

lxc launch images:debian/stretch vmtest

Creating vmtest
Starting vmtest

lxc storage volume create default myvolume size=10GB

Storage volume myvolume created

lxc storage volume attach default myvolume vmtest /data

zfs mount | grep myvolume

default/custom/myvolume /var/lib/lxd/storage-pools/default/custom/myvolume
default/custom/myvolume /var/lib/lxd/devices/vmtest/disk.data

Now the volume is mounted inside the CT and on the host here: /var/lib/lxd/storage-pools/default/custom/myvolume .
I can access data both side.

BUT

On 3.14 with snap now:

lxc launch images:debian/stretch vmtest

Creating vmtest
Starting vmtest

lxc storage volume create default myvolume size=10GB

Storage volume myvolume created

lxc storage volume attach default myvolume vmtest /data

zfs mount | grep myvolume

(empty)

zfs mount default/custom/myvolume

filesystem ‘default/custom/myvolume’ is already mounted
cannot mount ‘default/custom/myvolume’: mountpoint or dataset is busy

The volume is mounted inside the CT, but NOT on the host.

Am I doing something bad ?
Any help appreciated.

That’s likely because of the switch to the snap.
The LXD snap maintains a mount namespace to keep things hidden away from the host. This improves performance by keeping the mount table shorter, avoids confusion for various pieces of userspace software and makes it more likely that we can cleanly shutdown and delete containers without there being some open file blocking things.

You can see the inside of the LXD mount namespace through /var/snap/lxd/common/mntns, so when things are mounted, you should see them inside /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/custom/.

Wow, what a path :slight_smile:
Seems you saved my day here, thank you Stéphane.

Using this path I face a permission problem:

sudo -u lxdroot mkdir /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/custom/vm647566/test

mkdir: cannot create directory ‘/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/custom/vm647566/test’: Permission denied

namei -l /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/custom/vm647566/

f: /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/custom/vm647566/
drwxr-xr-x root root /
drwxr-xr-x root root var
drwxr-xr-x root root snap
drwxr-xr-x root root lxd
drwxr-xr-x root root common
lrwxrwxrwx root root mntns -> /proc/7040/root
drwxr-xr-x root root /
dr-xr-xr-x root root proc
dr-xr-xr-x root root 7040
lrwxrwxrwx root root root -> /
drwxr-xr-x root root /
drwxr-xr-x root root var
drwxr-xr-x root root snap
drwxr-xr-x root root lxd
drwxr-xr-x root root common
drwxr-xr-x root root lxd
drwx–x--x root root storage-pools
drwx–x--x root root default
drwx–x--x root root custom
drwxr-xr-x lxdroot lxdroot vm647566

If I “cd” (with root user) into the path, it work:

cd /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/custom/vm647566/

sudo -u lxdroot mkdir test

ls test

test

The problem since to be the access on /proc/7040/root/ (symlink to /) :
root@cpu-5195:~# sudo -u lxdroot ls /proc/7040/root/
ls: cannot access ‘/proc/7040/root/’: Permission denied
root@cpu-5195:~# sudo -u lxdroot ls /proc/7040/
attr coredump_filter gid_map mountinfo oom_score schedstat status
autogroup cpuset io mounts oom_score_adj sessionid syscall
auxv cwd limits mountstats pagemap setgroups task
cgroup environ loginuid net personality smaps timers
clear_refs exe map_files ns projid_map stack timerslack_ns
cmdline fd maps numa_maps root stat uid_map
comm fdinfo mem oom_adj sched statm wchan

sudo -u lxdroot ls /

bin data etc initrd.img lib media opt root sbin srv tmp var vmlinuz.old
boot dev home initrd.img.old lib64 mnt proc run snap sys usr vmlinuz

any clue ?

It seems that symlinks on /proc are not usual symlinks.

Any other solution than using this path over /proc ?

I wonder if using snap is a good solution.
Really don’t like how it brings layers under lxd :frowning: