ZFS mount point disappeared but LXC working

LXD
1

I have issues, sometimes the ZFS mounts for some containers are just seem to be disappeared, others are still there, all containers are working without problems and I found, that a similar issue was discussed already in 2016 here: https://lists.linuxcontainers.org/pipermail/lxc-users/2016-May/011742.html

I determined the same effect also several times and usually I just restarted the ‘affected’ containers. But now I just want to find a better solution. I did try zfs mount -a without any success.

I describe my findings here (with two exactly similar) containers
webhosting2 (mounts are ok)
webhosting3 (nothing seems to be mounted)

lxc list webhosting
+-------------+---------+----------------------+------+------------+-----------+
|    NAME     |  STATE  |         IPV4         | IPV6 |    TYPE    | SNAPSHOTS |
+-------------+---------+----------------------+------+------------+-----------+
| webhosting2 | RUNNING | 10.205.65.77 (eth0)  |      | PERSISTENT | 2         |
+-------------+---------+----------------------+------+------------+-----------+
| webhosting3 | RUNNING | 10.205.65.212 (eth0) |      | PERSISTENT | 0         |
+-------------+---------+----------------------+------+------------+-----------+

As said both are functional working. I can login to both. zfs list shows the mount points:

zfs list | grep containers/webhosting
lxd/containers/webhosting2   14.6G  1.00T  14.4G  /var/lib/lxd/storage-pools/lxd/containers/webhosting2
lxd/containers/webhosting3   1.33G  1.00T  1.61G  /var/lib/lxd/storage-pools/lxd/containers/webhosting3

But mount does only list the mountpoint for webhosting2:

mount | grep webhosting
lxd/containers/webhosting2 on /var/lib/lxd/storage-pools/lxd/containers/webhosting2 type zfs

zfs mountpoints are good:

zfs get mountpoint lxd/containers/webhosting2 lxd/containers/webhosting3
NAME                        PROPERTY    VALUE                                                  SOURCE
lxd/containers/webhosting2  mountpoint  /var/lib/lxd/storage-pools/lxd/containers/webhosting2  local
lxd/containers/webhosting3  mountpoint  /var/lib/lxd/storage-pools/lxd/containers/webhosting3  local

Please give me some hint, what else I can try. I need a solution to get the ZFS mountpoint back without rebooting the container. Please aks, what else I should provide to find a solution. I think I am not the only one who suffer on this problem.

lxc --version
2.21

@stgraber: I really appreciate your diligence, hard and good work - maybe you can give me a hint?
But this zfs mount -a does not help as you told 2 years ago.

2

I was going to ask the same kind of question, but found yours.
After setting up LXD with a previously created ZFS dataset as storage pool, I get the same behavior.

I’m using the LXD snap 3.7 on an ubuntu 18.04 host.

After launching a new container, the container seems to work perfectly, but its associated dataset is not mounted so I cannot inspect its rootfs from the host :

# lxc list
+------------+---------+---------------------+------+------------+-----------+
|    NAME    |  STATE  |        IPV4         | IPV6 |    TYPE    | SNAPSHOTS |
+------------+---------+---------------------+------+------------+-----------+
| bionictest | RUNNING | 10.10.10.235 (eth0) |      | PERSISTENT |           |
+------------+---------+---------------------+------+------------+-----------+
# lxc storage list
+-------+-------------+--------+------------------+---------+
| NAME  | DESCRIPTION | DRIVER |      SOURCE      | USED BY |
+-------+-------------+--------+------------------+---------+
| pool1 |             | zfs    | tank/lxd-storage | 3       |
+-------+-------------+--------+------------------+---------+

# zfs list  | grep bionictest
tank/lxd-storage/containers/bionictest                                                    37.6M  1.75T   788M  /var/snap/lxd/common/lxd/storage-pools/pool1/containers/bionictest

# mountpoint /var/snap/lxd/common/lxd/storage-pools/pool1/containers/bionictest
/var/snap/lxd/common/lxd/storage-pools/pool1/containers/bionictest is not a mountpoint

As @ingobaab reports, running zfs mount -a has no effect.

Instead if I explicitelty mount the container dataset with zfs mount tank/lxd-storage/containers/bionictest then I can see the mount.

# mountpoint /var/snap/lxd/common/lxd/storage-pools/pool1/containers/bionictest
/var/snap/lxd/common/lxd/storage-pools/pool1/containers/bionictest is a mountpoint
# ll /var/snap/lxd/common/lxd/storage-pools/pool1/containers/bionictest
total 49
drwxr-x--x+  4 1000000 1000000    6 Dec 13 03:33 ./
drwx--x--x   3 root    root    4096 Dec 13 03:33 ../
-r--------   1 root    root    2131 Dec 13 03:39 backup.yaml
-rw-r--r--   1 root    root    1048 Dec  6 18:05 metadata.yaml
drwxr-xr-x  22 1000000 1000000   22 Dec  6 17:02 rootfs/
drwxr-xr-x   2 root    root       7 Dec  6 18:05 templates/

I like to have the containers rootfs accessible on the hosts so I can look around and compare stuff in different containers easily from the hosts.

Is it normal/expected behavior to run a container without its dataset mounted on the host ?
What is the proper way to have the containers dataset mounted when the system boots ?

Thanks!