Can you nsenter on the lxd process rather than the lxc monitor process please (I think they are in same mountns but worth being sure anyway).
The PID returned from lxc info | grep server_pid
Also do you see the mount of the container on the LXD host normally (outside of the LXD package’s mount namespace)?
Also, take a look at a similar issue:
If you stop the container, then run the suggested steps in that post it should show what is still keeping the mount open and allow you to unmount it.
root@prod-us-west1-b-001:~# lxc info | grep server_pid
server_pid: 18044
root@prod-us-west1-b-001:~# nsenter -t 18044 -m mount | grep north-dion
default/containers/north-dion on /var/snap/lxd/common/shmounts/storage-pools/default/containers/north-dion type zfs (rw,xattr,posixacl)
1 Like
I run the commands but as shown in output there are no folders in this path in my installation:
This is blank
/var/snap/lxd/common/shmounts/
All storage pools are defined here instead:
/var/snap/lxd/common/lxd/storage-pools/default/containers/
Have you tried unmounting (after stopping the container) and then trying to start it?
nsenter -t PID -m umount /var/snap/lxd/common/shmounts/storage-pools/default/containers/north-dion
Yes it shows this:
nsenter -t 18044 -m umount /var/snap/lxd/common/shmounts/storage-pools/default/containers/north-dion
umount: /var/snap/lxd/common/shmounts/storage-pools/default/containers/north-dion: no mount point specified.
I think this is because my storage pools are defined here actually:
/var/snap/lxd/common/lxd/storage-pools/default/containers/north-dion
I tried this and it is saying no points:
nsenter -t 18044 -m umount /var/snap/lxd/common/lxd/storage-pools/default/containers/north-dion
umount: /var/snap/lxd/common/lxd/storage-pools/default/containers/north-dion: not mounted.
Thats normal where your storage pools are. But also your /var/snap/lxd/common/shmounts directory seems to have become out of sync.
I’m not familiar with how the /var/snap/lxd/common/shmounts directory works.
However there is a mention of a similar issue in the past here:
Also see
What about re-creating the directory and removing the symlinks I suggested?
Yes I tried creating the directory /var/snap/lxd/common/shmounts/instances and then removing the symbolic link /var/snap/lxd/common/lxd/shmounts . but it is not working. I didn’t run the snap lxd restart or any other commands as I am afraid my other running containers will stop.
OK well I think we will have to wait until @stgraber comes online to help with this as I don’t know what else to suggest I’m afraid. It seems the snap package’s mount namespace has become confused.
The unmount often falls in this case due to overmounting.
Try doing a nsenter -t PID umount -l /var/snap/lxd/common/shmounts
Followed with a umount of the problematic dataset.
We’ve seen it before and my best guess is some interaction between a core snap refresh and a LXD snap refresh, I’m still working on a reliable reproducer so I can then investigate what’s wrong with our mount reshuffling workaround.
2 Likes