Requesting for Help: LXD 3.23 zfs mount fails - filesystem already mounted

mohananpg · April 1, 2020, 5:09am

This is from NITK Surathkal, India. We request for help to resolve an issue.
We have servers running Ubuntu 18.04.4 LTS with

snap list
Name  Version    Rev    Tracking       Publisher   Notes
core  16-2.44.1  8935   latest/stable  canonical✓  core
lxd   3.23       14133  latest/stable  canonical✓  -

In some of the important containers, we get the following error

lxc snapshot NAME
Error: Create instance snapshot (mount source): Failed to run: zfs mount default/containers/NAME: cannot mount 'default/containers/NAME': filesystem already mounted

The snap was autoupgraded to the latest one and since then, we face this issue.
After stopping a container using lxc stop NAME, we get an error when we restart it.

lxc start NAME: Common start logic: Failed to run: zfs mount default/containers/NAME: cannot mount 'default/containers/NAME': filesystem already mounted

Could anyone help, please ?
Any pointers , previous cases, work arounds?
Thanks in advance.

Sincerely, Yours
MohananPG
NITK Surathkal, India

stgraber · April 1, 2020, 1:42pm

Chances are the problem is that the filesystem is already mounted

With the container stopped, run grep default/containers/NAME /proc/*/mountinfo, that should show you where it’s already mounted and for what processes.

You can then use nsenter -t PID -m umount /path/to/the/mount

mohananpg · April 2, 2020, 6:19am

Thank you very much. It worked!

leopaul36 · April 3, 2020, 12:59pm

Hello @stgraber,

I just encountered the same issue so I figured I should post here.

I am unable to start a container with the same error:

root@n3:~# lxc start au-kfk-brk-3
Error: Common start logic: Failed to run: zfs mount syspool/lxd/containers/au-kfk-brk-3: cannot mount 'syspool/lxd/containers/au-kfk-brk-3': filesystem already mounted
Try `lxc info --show-log au-kfk-brk-3` for more info

The fs is not mounted according to zfs:

root@n3:~# zfs get -r mounted syspool | grep au-kfk-brk-3
syspool/lxd/containers/au-kfk-brk-3                                                                         mounted   no       -

I tried to run grep syspool/lxd/containers/au-kfk-brk-3 /proc/*/mountinfo as you mentionned, but this is where I’m getting confused:

root@n3:~# grep syspool/lxd/containers/au-kfk-brk-3 /proc/*/mountinfo
/proc/10438/mountinfo:2765 839 0:238 / /var/snap/lxd/common/shmounts/storage-pools/system/containers/au-kfk-brk-3 rw shared:212 - zfs syspool/lxd/containers/au-kfk-brk-3 rw,xattr,posixacl
/proc/10920/mountinfo:2765 839 0:238 / /var/snap/lxd/common/shmounts/storage-pools/system/containers/au-kfk-brk-3 rw shared:212 - zfs syspool/lxd/containers/au-kfk-brk-3 rw,xattr,posixacl
/proc/12074/mountinfo:2765 839 0:238 / /var/snap/lxd/common/shmounts/storage-pools/system/containers/au-kfk-brk-3 rw shared:212 - zfs syspool/lxd/containers/au-kfk-brk-3 rw,xattr,posixacl
/proc/12553/mountinfo:2765 839 0:238 / /var/snap/lxd/common/shmounts/storage-pools/system/containers/au-kfk-brk-3 rw shared:212 - zfs syspool/lxd/containers/au-kfk-brk-3 rw,xattr,posixacl
/proc/13619/mountinfo:2765 839 0:238 / /var/snap/lxd/common/shmounts/storage-pools/system/containers/au-kfk-brk-3 rw shared:212 - zfs syspool/lxd/containers/au-kfk-brk-3 rw,xattr,posixacl
/proc/14078/mountinfo:2765 839 0:238 / /var/snap/lxd/common/shmounts/storage-pools/system/containers/au-kfk-brk-3 rw shared:212 - zfs syspool/lxd/containers/au-kfk-brk-3 rw,xattr,posixacl
/proc/14461/mountinfo:2765 839 0:238 / /var/snap/lxd/common/shmounts/storage-pools/system/containers/au-kfk-brk-3 rw shared:212 - zfs syspool/lxd/containers/au-kfk-brk-3 rw,xattr,posixacl
/proc/16301/mountinfo:2765 839 0:238 / /var/snap/lxd/common/shmounts/storage-pools/system/containers/au-kfk-brk-3 rw shared:212 - zfs syspool/lxd/containers/au-kfk-brk-3 rw,xattr,posixacl
...

I have 39 different processes which are all the “lxc monitor” processes of every container on my machine (30 running + 9 stopped).

Here is one these process for example:

root@n3:~# ps aux | grep 34214
root     23348  0.0  0.0   7952  2684 pts/0    S+   12:54   0:00 grep --color=auto 34214
root     34214  0.0  0.0 291812 13332 ?        Ss   Mar04   0:00 [lxc monitor] /var/snap/lxd/common/lxd/containers au-rng-adm-1

Is it safe to run the umount on these processes? Will it affect the other running containers?

I am running LXD 4.0.0 Rev 14167.

Thanks for your help.

NicolasGoeddel · September 8, 2020, 6:38pm

I’ve got exactly the same issue as you had with LXD version 4.4 on Ubuntu 18.04.

There are two containers I can not start anymore after stopping them. It tells me

Error: Common start logic: Failed to run: zfs mount lxd/containers/vtiger: cannot mount 'lxd/containers/vtiger': filesystem already mounted
Try `lxc info --show-log vtiger` for more info

However I can start them if I mount the volume manually with zfs mount lxd/containers/vtiger. After that starting the container works without an issue. Now when I try to restart the container again I get this:

Error: Failed to run: /snap/lxd/current/bin/lxd forkstart vtiger /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/vtiger/lxc.conf: 
Try `lxc info --show-log vtiger` for more info

And there is also more:

Log:

lxc vtiger 20200908183407.624 WARN     cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.vtiger"
lxc vtiger 20200908183407.625 WARN     cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.vtiger"
lxc vtiger 20200908183407.626 ERROR    utils - utils.c:lxc_can_use_pidfd:1834 - Kernel does not support pidfds
lxc vtiger 20200908183407.709 ERROR    dir - storage/dir.c:dir_mount:152 - No such file or directory - Failed to mount "/var/snap/lxd/common/lxd/containers/vtiger/rootfs" on "/var/snap/lxd/common/lxc/"
lxc vtiger 20200908183407.710 ERROR    conf - conf.c:lxc_mount_rootfs:1257 - Failed to mount rootfs "/var/snap/lxd/common/lxd/containers/vtiger/rootfs" onto "/var/snap/lxd/common/lxc/" with options "(null)"
lxc vtiger 20200908183407.710 ERROR    conf - conf.c:lxc_setup_rootfs_prepare_root:3094 - Failed to setup rootfs for
lxc vtiger 20200908183407.710 ERROR    conf - conf.c:lxc_setup:3194 - Failed to setup rootfs
lxc vtiger 20200908183407.710 ERROR    start - start.c:do_start:1224 - Failed to setup container "vtiger"
lxc vtiger 20200908183407.711 ERROR    sync - sync.c:__sync_wait:41 - An error occurred in another process (expected sequence number 5)
lxc vtiger 20200908183407.711 WARN     network - network.c:lxc_delete_network_priv:3185 - Failed to rename interface with index 2 from "eth0" to its initial name "veth8142e3b8"
lxc vtiger 20200908183407.711 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:850 - Received container state "ABORTING" instead of "RUNNING"
lxc vtiger 20200908183407.711 ERROR    start - start.c:__lxc_start:1999 - Failed to spawn container "vtiger"
lxc vtiger 20200908183407.711 WARN     start - start.c:lxc_abort:1024 - No such process - Failed to send SIGKILL to 11465
lxc 20200908183407.858 WARN     commands - commands.c:lxc_cmd_rsp_recv:124 - Connection reset by peer - Failed to receive response for command "get_state"
lxc vtiger 20200908183418.457 ERROR    conf - conf.c:run_buffer:324 - Script exited with status 1
lxc vtiger 20200908183418.457 ERROR    start - start.c:lxc_end:965 - Failed to run lxc.hook.post-stop for container "vtiger"

After a zfs unmount lxd/containers/vtiger and another mount it works again. And there are many processes too which still seem to have mounted the volume:

# grep vtiger /proc/*/mountinfo
/proc/10111/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/11175/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/11259/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/11708/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/14331/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/17837/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/19686/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/2427/mountinfo:1823 552 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/25003/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/25906/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/2748/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/29450/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/29538/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/3480/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/3574/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/620/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/6323/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/6418/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/6663/mountinfo:1823 552 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/9778/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
/proc/9956/mountinfo:5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl

How can I solve the issue now? How did you solve it?

leopaul36 · September 8, 2020, 7:46pm

I ended running the nsenter -t PID -m umount /path/to/the/mount command which solved the issue.

NicolasGoeddel · September 9, 2020, 10:38am

It does not work. I tried both versions:

# nsenter -t 10111 -m umount /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger
umount: /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger: no mount point specified.

and

# nsenter -t 10111 -m umount lxd/containers/vtiger
umount: /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger: no mount point specified.

@stgraber Do you have any idea to solve this without restarting the host?

stgraber · September 9, 2020, 7:31pm

Can you show the whole content of /proc/10111/mountinfo?

The failure from your earlier umount attempt suggests the path got overmounted somehow.

NicolasGoeddel · September 9, 2020, 9:41pm

This is /proc/10111/mountinfo (modified to cover internal project names):

136 5668 9:1 / /var/lib/snapd/hostfs rw,relatime master:1 - ext4 /dev/md1 rw,data=ordered
396 136 0:23 / /var/lib/snapd/hostfs/run rw,nosuid,noexec,relatime master:5 - tmpfs tmpfs rw,size=6591820k,mode=755
442 396 0:26 / /var/lib/snapd/hostfs/run/lock rw,nosuid,nodev,noexec,relatime master:6 - tmpfs tmpfs rw,size=5120k
511 396 0:23 /snapd/ns /var/lib/snapd/hostfs/run/snapd/ns rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=6591820k,mode=755
520 396 0:188 / /var/lib/snapd/hostfs/run/user/0 rw,nosuid,nodev,relatime master:1287 - tmpfs tmpfs rw,size=6591816k,mode=700
5406 136 7:1 / /var/lib/snapd/hostfs/snap/core18/1880 ro,nodev,relatime master:32 - squashfs /dev/loop1 ro
5409 136 7:5 / /var/lib/snapd/hostfs/snap/core/9665 ro,nodev,relatime master:37 - squashfs /dev/loop5 ro
5410 136 9:0 / /var/lib/snapd/hostfs/boot rw,relatime master:38 - ext4 /dev/md0 rw,data=ordered
5411 136 0:45 / /var/lib/snapd/hostfs/home rw master:39 - zfs lxd/home rw,xattr,posixacl
5508 136 0:50 / /var/lib/snapd/hostfs/var/lib/lxcfs rw,nosuid,nodev,relatime master:192 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
5509 136 0:53 / /var/lib/snapd/hostfs/var/snap/lxd/common/ns rw,relatime - tmpfs tmpfs rw,size=1024k,mode=700
5511 136 7:4 / /var/lib/snapd/hostfs/snap/core18/1885 ro,nodev,relatime master:1470 - squashfs /dev/loop4 ro
5580 92 7:4 / / ro,nodev,relatime master:1470 - squashfs /dev/loop4 ro
5581 5580 0:6 / /dev rw,nosuid,relatime master:2 - devtmpfs udev rw,size=32927560k,nr_inodes=8231890,mode=755
5582 5581 0:22 / /dev/pts rw,nosuid,noexec,relatime master:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
5583 5581 0:25 / /dev/shm rw,nosuid,nodev master:4 - tmpfs tmpfs rw
5584 5581 0:19 / /dev/mqueue rw,relatime master:27 - mqueue mqueue rw
5585 5581 0:43 / /dev/hugepages rw,relatime master:28 - hugetlbfs hugetlbfs rw,pagesize=2M
5587 5580 0:45 / /home rw master:39 - zfs lxd/home rw,xattr,posixacl
5588 5580 9:1 /root /root rw,relatime master:1 - ext4 /dev/md1 rw,data=ordered
5589 5580 0:4 / /proc rw,nosuid,nodev,noexec,relatime master:13 - proc proc rw
5590 5589 0:42 / /proc/sys/fs/binfmt_misc rw,relatime master:25 - autofs systemd-1 rw,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19697
5591 5590 0:68 / /proc/sys/fs/binfmt_misc rw,relatime master:257 - binfmt_misc binfmt_misc rw
5592 5580 0:21 / /sys rw,nosuid,nodev,noexec,relatime master:7 - sysfs sysfs rw
5593 5592 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime master:8 - securityfs securityfs rw
5609 5592 0:27 / /sys/fs/cgroup ro,nosuid,nodev,noexec master:9 - tmpfs tmpfs ro,mode=755
5610 5609 0:28 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime master:10 - cgroup2 cgroup rw
5611 5609 0:29 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime master:11 - cgroup cgroup rw,xattr,name=systemd
5612 5609 0:31 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime master:14 - cgroup cgroup rw,net_cls,net_prio
5613 5609 0:32 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime master:15 - cgroup cgroup rw,rdma
5614 5609 0:33 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime master:16 - cgroup cgroup rw,devices
5615 5609 0:34 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime master:17 - cgroup cgroup rw,blkio
5616 5609 0:35 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime master:18 - cgroup cgroup rw,pids
5617 5609 0:36 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime master:19 - cgroup cgroup rw,freezer
5618 5609 0:37 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime master:20 - cgroup cgroup rw,memory
5619 5609 0:38 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime master:21 - cgroup cgroup rw,perf_event
5620 5609 0:39 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime master:22 - cgroup cgroup rw,cpu,cpuacct
5621 5609 0:40 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime master:23 - cgroup cgroup rw,cpuset,clone_children
5622 5609 0:41 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime master:24 - cgroup cgroup rw,hugetlb
5623 5592 0:30 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime master:12 - pstore pstore rw
5624 5592 0:8 / /sys/kernel/debug rw,relatime master:26 - debugfs debugfs rw
5625 5624 0:11 / /sys/kernel/debug/tracing rw,relatime master:29 - tracefs tracefs rw
5626 5592 0:44 / /sys/fs/fuse/connections rw,relatime master:30 - fusectl fusectl rw
5627 5592 0:20 / /sys/kernel/config rw,relatime master:31 - configfs configfs rw
5628 5580 9:1 /tmp /tmp rw,relatime master:1 - ext4 /dev/md1 rw,data=ordered
5629 5580 9:1 /var/snap /var/snap rw,relatime master:1 - ext4 /dev/md1 rw,data=ordered
5630 5629 0:53 / /var/snap/lxd/common/ns rw,relatime - tmpfs tmpfs rw,size=1024k,mode=700
5631 5580 9:1 /var/lib/dhcp /var/lib/dhcp rw,relatime master:1 - ext4 /dev/md1 rw,data=ordered
5632 5580 9:1 /var/lib/snapd /var/lib/snapd rw,relatime master:1 - ext4 /dev/md1 rw,data=ordered
5633 5580 9:1 /var/tmp /var/tmp rw,relatime master:1 - ext4 /dev/md1 rw,data=ordered
5638 5580 9:1 /lib/modules /lib/modules rw,relatime master:1 - ext4 /dev/md1 rw,data=ordered
5639 5580 9:1 /lib/firmware /lib/firmware rw,relatime master:1 - ext4 /dev/md1 rw,data=ordered
5640 5580 9:1 /usr/src /usr/src rw,relatime master:1 - ext4 /dev/md1 rw,data=ordered
5641 5580 9:1 /var/log /var/log rw,relatime master:1 - ext4 /dev/md1 rw,data=ordered
5642 5580 9:1 /media /media rw,relatime shared:1 - ext4 /dev/md1 rw,data=ordered
5644 5580 9:1 /mnt /mnt rw,relatime master:1 - ext4 /dev/md1 rw,data=ordered
5647 5580 7:5 /usr/lib/snapd /usr/lib/snapd ro,nodev,relatime master:37 - squashfs /dev/loop5 ro
5648 5580 9:1 /snap /snap rw,relatime master:1 - ext4 /dev/md1 rw,data=ordered
5649 5648 7:1 / /snap/core18/1880 ro,nodev,relatime master:32 - squashfs /dev/loop1 ro
5652 5648 7:5 / /snap/core/9665 ro,nodev,relatime master:37 - squashfs /dev/loop5 ro
5654 5648 7:4 / /snap/core18/1885 ro,nodev,relatime master:1470 - squashfs /dev/loop4 ro
5668 5632 9:1 /var/lib/snapd/hostfs /var/lib/snapd/hostfs rw,relatime - ext4 /dev/md1 rw,data=ordered
175 5628 9:1 /tmp/snap.lxd/tmp /tmp rw,relatime - ext4 /dev/md1 rw,data=ordered
217 5582 0:197 / /dev/pts rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
256 5581 0:197 /ptmx /dev/ptmx rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
5024 136 7:2 / /var/lib/snapd/hostfs/snap/core/9804 ro,nodev,relatime master:1411 - squashfs /dev/loop2 ro
4959 5648 7:2 / /snap/core/9804 ro,nodev,relatime master:1411 - squashfs /dev/loop2 ro
5368 5629 9:1 /var/snap/lxd/common/lxd/storage-pools /var/snap/lxd/common/lxd/storage-pools rw,relatime shared:1493 master:1 - ext4 /dev/md1 rw,data=ordered
5369 5629 9:1 /var/snap/lxd/common/lxd/devices /var/snap/lxd/common/lxd/devices rw,relatime shared:1494 master:1 - ext4 /dev/md1 rw,data=ordered
5680 5629 0:52 / /var/snap/lxd/common/shmounts rw,relatime shared:204 - tmpfs tmpfs rw,size=1024k,mode=711
5681 5680 0:59 / /var/snap/lxd/common/shmounts/lxcfs rw,nosuid,nodev,relatime shared:201 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
5682 5680 0:61 / /var/snap/lxd/common/shmounts/containers rw,relatime shared:263 - tmpfs tmpfs rw,size=100k,mode=711
5375 5680 0:54 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/belbooks rw shared:199 - zfs lxd/containers/belbooks rw,xattr,posixacl
5395 5680 0:115 / /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger rw shared:603 - zfs lxd/containers/vtiger rw,xattr,posixacl
603 5368 0:77 / /var/snap/lxd/common/lxd/storage-pools/lxd/containers/bfd-pubsearch rw shared:83 - zfs lxd/containers/bfd-pubsearch rw,xattr,posixacl
390 5680 0:194 / /var/snap/lxd/common/shmounts rw,relatime shared:823 - tmpfs tmpfs rw,size=1024k,mode=711
5372 5580 7:0 /wrappers/kmod /bin/kmod ro,nodev,relatime - squashfs /dev/loop0 ro
5378 5580 9:1 /usr/share/ca-certificates /usr/share/ca-certificates rw,relatime master:1 - ext4 /dev/md1 rw,data=ordered
5380 136 0:6 / /var/lib/snapd/hostfs/dev rw,nosuid,relatime master:2 - devtmpfs udev rw,size=32927560k,nr_inodes=8231890,mode=755
5381 136 0:4 / /var/lib/snapd/hostfs/proc rw,nosuid,nodev,noexec,relatime master:13 - proc proc rw
5382 136 0:21 / /var/lib/snapd/hostfs/sys rw,nosuid,nodev,noexec,relatime master:7 - sysfs sysfs rw
5384 390 0:202 / /var/snap/lxd/common/shmounts/lxcfs rw,nosuid,nodev,relatime shared:1402 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
5387 390 0:203 / /var/snap/lxd/common/shmounts/instances rw,relatime shared:1495 - tmpfs tmpfs rw,size=100k,mode=711
5390 5629 0:204 / /var/snap/lxd/common/lxd/devlxd rw,relatime - tmpfs tmpfs rw,size=100k,mode=755
4693 136 7:7 / /var/lib/snapd/hostfs/snap/lxd/16889 ro,nodev,relatime master:1171 - squashfs /dev/loop7 ro
4685 5648 7:7 / /snap/lxd/16889 ro,nodev,relatime master:1171 - squashfs /dev/loop7 ro
4439 136 7:8 / /var/lib/snapd/hostfs/snap/lxd/16926 ro,nodev,relatime master:1115 - squashfs /dev/loop8 ro
4431 5648 7:8 / /snap/lxd/16926 ro,nodev,relatime master:1115 - squashfs /dev/loop8 ro
4442 5580 9:0 / /boot rw,relatime master:38 - ext4 /dev/md0 rw,data=ordered
4443 5580 0:195 / /run rw,nosuid,nodev,relatime - tmpfs tmpfs rw,mode=755
4444 5580 0:196 / /etc rw,relatime - tmpfs tmpfs rw,mode=755
4445 5580 0:198 / /usr/share/misc rw,nosuid,nodev,relatime - tmpfs tmpfs rw,mode=755
4196 5368 0:189 / /var/snap/lxd/common/lxd/storage-pools/lxd/containers/sonarqube rw shared:1122 - zfs lxd/containers/sonarqube rw,xattr,posixacl
4771 136 7:9 / /var/lib/snapd/hostfs/snap/bashtop/128 ro,nodev,relatime master:820 - squashfs /dev/loop9 ro
4761 5648 7:9 / /snap/bashtop/128 ro,nodev,relatime master:820 - squashfs /dev/loop9 ro
607 5368 0:63 / /var/snap/lxd/common/lxd/storage-pools/lxd/containers/swcloud rw,noatime shared:33 - zfs lxd/containers/swcloud rw,xattr,posixacl
1110 5368 0:115 / /var/snap/lxd/common/lxd/storage-pools/lxd/containers/vtiger rw,noatime shared:394 master:389 - zfs lxd/containers/vtiger rw,xattr,posixacl
1109 5629 0:115 / /var/snap/lxd/common/lxd/storage-pools/lxd/containers/vtiger rw,noatime master:389 - zfs lxd/containers/vtiger rw,xattr,posixacl
1108 136 0:115 / /var/lib/snapd/hostfs/var/snap/lxd/common/lxd/storage-pools/lxd/containers/vtiger rw,noatime master:389 - zfs lxd/containers/vtiger rw,xattr,posixacl
94 5368 0:57 / /var/snap/lxd/common/lxd/storage-pools/lxd/containers/book rw,noatime shared:34 - zfs lxd/containers/book rw,xattr,posixacl
546 5368 0:74 / /var/snap/lxd/common/lxd/storage-pools/lxd/containers/db rw,noatime shared:35 - zfs lxd/containers/db rw,xattr,posixacl
724 5368 0:75 / /var/snap/lxd/common/lxd/storage-pools/lxd/containers/pubsession rw,noatime shared:36 - zfs lxd/containers/pubsession rw,xattr,posixacl
533 5368 0:111 / /var/snap/lxd/common/lxd/storage-pools/lxd/containers/docker rw,noatime shared:75 - zfs lxd/containers/docker rw,xattr,posixacl
2316 5368 0:117 / /var/snap/lxd/common/lxd/storage-pools/lxd/containers/gitlab rw,noatime shared:323 - zfs lxd/containers/gitlab rw,xattr,posixacl
2039 5368 0:120 / /var/snap/lxd/common/lxd/storage-pools/lxd/containers/w2020 rw,noatime shared:376 - zfs lxd/containers/w2020 rw,xattr,posixacl
88 5368 0:116 / /var/snap/lxd/common/lxd/storage-pools/lxd/containers/matomo rw,noatime shared:252 - zfs lxd/containers/matomo rw,xattr,posixacl
594 5368 0:137 / /var/snap/lxd/common/lxd/storage-pools/lxd/containers/test rw,noatime shared:545 - zfs lxd/containers/test rw,xattr,posixacl
91 5368 0:114 / /var/snap/lxd/common/lxd/storage-pools/lxd/containers/bird rw,noatime shared:546 - zfs lxd/containers/bird rw,xattr,posixacl

stgraber · September 9, 2020, 9:54pm

Ok, so maybe try:

nsenter -t 10111 umount -l /var/snap/lxd/common/shmounts
nsenter -t 10111 umount -l /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger

NicolasGoeddel · September 11, 2020, 3:52pm

Unfortunately no luck:

# nsenter -t 10111 umount -l /var/snap/lxd/common/shmounts
umount: /var/snap/lxd/common/shmounts: not mounted.
# nsenter -t 10111 umount -l /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger
umount: /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger: no mount point specified.

Edit:
Wait. This worked:

nsenter -t 10111 -m umount -l /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/vtiger

Now grep lxd/containers/vtiger /proc/*/mountinfo does returning nothing.

But when I now try to start the container again I get a new error:

# lxc start vtiger
Error: Common start logic: mkdir /var/snap/lxd/common/lxd/shmounts: file exists

Further investigation shows this:

# readlink /var/snap/lxd/common/lxd/shmounts
/var/snap/lxd/common/shmounts/instances
# stat /var/snap/lxd/common/shmounts/instances
stat: cannot stat '/var/snap/lxd/common/shmounts/instances': No such file or directory

Is this because I also tried to unmount /var/snap/lxd/common/shmounts?

stgraber · September 11, 2020, 9:13pm

Ah yeah, this is getting a bit messy…
Easy workaround would be to create the missing directory under shmounts.

NicolasGoeddel · September 12, 2020, 8:39am

I was able to make it work again by first creating the directory /var/snap/lxd/common/shmounts/instances and then removing the symbolic link /var/snap/lxd/common/lxd/shmounts.
Now all the containers start again.