Problem with lxc: Failed initializing storage pool "default"

khalid_mrabti · August 11, 2021, 6:25pm

hello everyone,
I have a problem with lxc this morning and searched for a solution for hours, now this is my last hope.
the output: # lxc list
is : Error: Get “http://unix.socket/1.0”: EOF
the only commanded starting with lxc and works is lxc --help (always the same error)
when running #lxd --debug
the output is this error:
Error: Failed initializing storage pool “default”: Failed to mount “/dev/loop2” on “/var/snap/lxd/common/lxd/storage-pools/default” using “btrfs”: file exists

output of # ls /var/snap/lxd/common/lxd/storage-pools/
default rdv-storage-pool

tomp · August 11, 2021, 6:36pm

Is the default directory empty?

khalid_mrabti · August 12, 2021, 11:18am

hello @tomp and thanks for replaying
yes it is empty
#ls /var/snap/lxd/common/lxd/storage-pools/default
returns nothing

tomp · August 12, 2021, 11:54am

Try doing rmdir /var/snap/lxd/common/lxd/storage-pools/default then and see if that allows you to start LXD.

khalid_mrabti · August 12, 2021, 12:07pm

hello @tomp and thanks for replaying again, now when I deleted the folder, the error is no such file or directory
the debug output :

tomp · August 12, 2021, 1:00pm

OK if you recreate the directory again, then can you check if it shows in cat /proc/mounts

khalid_mrabti · August 12, 2021, 1:11pm

there is no difference and, the error remains.
And for your question the answer is no, I don’t see that it is mounted
the output of #cat /proc/mounts is:

tomp · August 12, 2021, 1:17pm

Have you tried rebooting so that all of the loop mounts and snap package mounts are reset?

khalid_mrabti · August 12, 2021, 1:19pm

no, I didn’t, the machine is a production server, Am afraid that rebooting the system causes a shutdown of the containers, so everything goes down

tomp · August 12, 2021, 1:21pm

What do you see in:

sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- mount

khalid_mrabti · August 12, 2021, 1:28pm

this is the output as image:

as text:

/dev/md2 on /var/lib/snapd/hostfs type ext4 (rw,relatime)
tmpfs on /var/lib/snapd/hostfs/run type tmpfs (rw,nosuid,noexec,relatime,size=6585868k,mode=755)
tmpfs on /var/lib/snapd/hostfs/run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /var/lib/snapd/hostfs/run/snapd/ns type tmpfs (rw,nosuid,noexec,relatime,size=6585868k,mode=755)
tmpfs on /var/lib/snapd/hostfs/run/user/110 type tmpfs (rw,nosuid,nodev,relatime,size=6585864k,mode=700,uid=110,gid=117)
/dev/sda1 on /var/lib/snapd/hostfs/boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
/dev/md4 on /var/lib/snapd/hostfs/opt type ext4 (rw,relatime)
/var/lib/snapd/snaps/core20_1026.snap on /var/lib/snapd/hostfs/snap/core20/1026 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core18_2074.snap on /var/lib/snapd/hostfs/snap/core18/2074 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core_11316.snap on /var/lib/snapd/hostfs/snap/core/11316 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/certbot_1280.snap on /var/lib/snapd/hostfs/snap/certbot/1280 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/lxd_21039.snap on /var/lib/snapd/hostfs/snap/lxd/21039 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core20_1081.snap on /var/lib/snapd/hostfs/snap/core20/1081 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core_11420.snap on /var/lib/snapd/hostfs/snap/core/11420 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/certbot_1343.snap on /var/lib/snapd/hostfs/snap/certbot/1343 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/lxd_21260.snap on /var/lib/snapd/hostfs/snap/lxd/21260 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core18_2128.snap on /var/lib/snapd/hostfs/snap/core18/2128 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core20_1081.snap on / type squashfs (ro,nodev,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=32912512k,nr_inodes=8228128,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
/dev/md2 on /etc type ext4 (rw,relatime)
/dev/md2 on /home type ext4 (rw,relatime)
/dev/md2 on /root type ext4 (rw,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11231)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset,clone_children)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/md2 on /tmp type ext4 (rw,relatime)
/dev/md2 on /var/snap type ext4 (rw,relatime)
/dev/md2 on /var/lib/snapd type ext4 (rw,relatime)
/dev/md2 on /var/tmp type ext4 (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=6585868k,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,noexec,relatime,size=6585868k,mode=755)
tmpfs on /run/user/110 type tmpfs (rw,nosuid,nodev,relatime,size=6585864k,mode=700,uid=110,gid=117)
/dev/md2 on /usr/lib/modules type ext4 (rw,relatime)
/dev/md2 on /usr/lib/firmware type ext4 (rw,relatime)
/dev/md2 on /usr/src type ext4 (rw,relatime)
/dev/md2 on /var/log type ext4 (rw,relatime)
/dev/md2 on /media type ext4 (rw,relatime)
tmpfs on /run/netns type tmpfs (rw,nosuid,noexec,relatime,size=6585868k,mode=755)
/dev/md2 on /mnt type ext4 (rw,relatime)
/var/lib/snapd/snaps/core20_1081.snap on /etc/nsswitch.conf type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core20_1081.snap on /etc/apparmor type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core20_1081.snap on /etc/apparmor.d type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core_11420.snap on /usr/lib/snapd type squashfs (ro,nodev,relatime)
/dev/md2 on /snap type ext4 (rw,relatime)
/var/lib/snapd/snaps/core20_1026.snap on /snap/core20/1026 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core18_2074.snap on /snap/core18/2074 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core_11316.snap on /snap/core/11316 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/certbot_1280.snap on /snap/certbot/1280 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/lxd_21039.snap on /snap/lxd/21039 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core20_1081.snap on /snap/core20/1081 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core_11420.snap on /snap/core/11420 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/certbot_1343.snap on /snap/certbot/1343 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/lxd_21260.snap on /snap/lxd/21260 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core18_2128.snap on /snap/core18/2128 type squashfs (ro,nodev,relatime)
/dev/md2 on /var/lib/snapd/hostfs type ext4 (rw,relatime)
/dev/md2 on /tmp type ext4 (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/ptmx type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
tmpfs on /var/snap/lxd/common/lxd/devlxd type tmpfs (rw,relatime,size=100k,mode=755)
/dev/md2 on /var/snap/lxd/common/lxd/storage-pools type ext4 (rw,relatime)
/dev/md2 on /var/snap/lxd/common/lxd/devices type ext4 (rw,relatime)
tmpfs on /var/snap/lxd/common/shmounts type tmpfs (rw,relatime,size=1024k,mode=711)
/var/lib/snapd/snaps/lxd_21260.snap on /usr/bin/kmod type squashfs (ro,nodev,relatime)
/dev/md2 on /usr/share/ca-certificates type ext4 (rw,relatime)
udev on /var/lib/snapd/hostfs/dev type devtmpfs (rw,nosuid,relatime,size=32912512k,nr_inodes=8228128,mode=755)
proc on /var/lib/snapd/hostfs/proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /var/lib/snapd/hostfs/sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /var/snap/lxd/common/shmounts/instances type tmpfs (rw,relatime,size=100k,mode=711)
tmpfs on /run/user/998 type tmpfs (rw,nosuid,nodev,relatime,size=6585864k,mode=700,uid=998,gid=998)
tmpfs on /var/lib/snapd/hostfs/run/user/998 type tmpfs (rw,nosuid,nodev,relatime,size=6585864k,mode=700,uid=998,gid=998)
lxcfs on /var/snap/lxd/common/shmounts/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/md2 on /boot type ext4 (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
tmpfs on /etc type tmpfs (rw,relatime,mode=755)
tmpfs on /usr/share/misc type tmpfs (rw,relatime,mode=755)
root@ns3033493:~# vi nsenter_output.txt
root@ns3033493:~# nsenter --mount=/run/snapd/ns/lxd.mnt -- mount
/dev/md2 on /var/lib/snapd/hostfs type ext4 (rw,relatime)
tmpfs on /var/lib/snapd/hostfs/run type tmpfs (rw,nosuid,noexec,relatime,size=6585868k,mode=755)
tmpfs on /var/lib/snapd/hostfs/run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /var/lib/snapd/hostfs/run/snapd/ns type tmpfs (rw,nosuid,noexec,relatime,size=6585868k,mode=755)
tmpfs on /var/lib/snapd/hostfs/run/user/110 type tmpfs (rw,nosuid,nodev,relatime,size=6585864k,mode=700,uid=110,gid=117)
/dev/sda1 on /var/lib/snapd/hostfs/boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
/dev/md4 on /var/lib/snapd/hostfs/opt type ext4 (rw,relatime)
/var/lib/snapd/snaps/core20_1026.snap on /var/lib/snapd/hostfs/snap/core20/1026 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core18_2074.snap on /var/lib/snapd/hostfs/snap/core18/2074 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core_11316.snap on /var/lib/snapd/hostfs/snap/core/11316 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/certbot_1280.snap on /var/lib/snapd/hostfs/snap/certbot/1280 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/lxd_21039.snap on /var/lib/snapd/hostfs/snap/lxd/21039 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core20_1081.snap on /var/lib/snapd/hostfs/snap/core20/1081 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core_11420.snap on /var/lib/snapd/hostfs/snap/core/11420 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/certbot_1343.snap on /var/lib/snapd/hostfs/snap/certbot/1343 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/lxd_21260.snap on /var/lib/snapd/hostfs/snap/lxd/21260 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core18_2128.snap on /var/lib/snapd/hostfs/snap/core18/2128 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core20_1081.snap on / type squashfs (ro,nodev,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=32912512k,nr_inodes=8228128,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
/dev/md2 on /etc type ext4 (rw,relatime)
/dev/md2 on /home type ext4 (rw,relatime)
/dev/md2 on /root type ext4 (rw,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11231)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset,clone_children)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/md2 on /tmp type ext4 (rw,relatime)
/dev/md2 on /var/snap type ext4 (rw,relatime)
/dev/md2 on /var/lib/snapd type ext4 (rw,relatime)
/dev/md2 on /var/tmp type ext4 (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=6585868k,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,noexec,relatime,size=6585868k,mode=755)
tmpfs on /run/user/110 type tmpfs (rw,nosuid,nodev,relatime,size=6585864k,mode=700,uid=110,gid=117)
/dev/md2 on /usr/lib/modules type ext4 (rw,relatime)
/dev/md2 on /usr/lib/firmware type ext4 (rw,relatime)
/dev/md2 on /usr/src type ext4 (rw,relatime)
/dev/md2 on /var/log type ext4 (rw,relatime)
/dev/md2 on /media type ext4 (rw,relatime)
tmpfs on /run/netns type tmpfs (rw,nosuid,noexec,relatime,size=6585868k,mode=755)
/dev/md2 on /mnt type ext4 (rw,relatime)
/var/lib/snapd/snaps/core20_1081.snap on /etc/nsswitch.conf type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core20_1081.snap on /etc/apparmor type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core20_1081.snap on /etc/apparmor.d type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core_11420.snap on /usr/lib/snapd type squashfs (ro,nodev,relatime)
/dev/md2 on /snap type ext4 (rw,relatime)
/var/lib/snapd/snaps/core20_1026.snap on /snap/core20/1026 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core18_2074.snap on /snap/core18/2074 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core_11316.snap on /snap/core/11316 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/certbot_1280.snap on /snap/certbot/1280 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/lxd_21039.snap on /snap/lxd/21039 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core20_1081.snap on /snap/core20/1081 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core_11420.snap on /snap/core/11420 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/certbot_1343.snap on /snap/certbot/1343 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/lxd_21260.snap on /snap/lxd/21260 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core18_2128.snap on /snap/core18/2128 type squashfs (ro,nodev,relatime)
/dev/md2 on /var/lib/snapd/hostfs type ext4 (rw,relatime)
/dev/md2 on /tmp type ext4 (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/ptmx type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
tmpfs on /var/snap/lxd/common/lxd/devlxd type tmpfs (rw,relatime,size=100k,mode=755)
/dev/md2 on /var/snap/lxd/common/lxd/storage-pools type ext4 (rw,relatime)
/dev/md2 on /var/snap/lxd/common/lxd/devices type ext4 (rw,relatime)
tmpfs on /var/snap/lxd/common/shmounts type tmpfs (rw,relatime,size=1024k,mode=711)
/var/lib/snapd/snaps/lxd_21260.snap on /usr/bin/kmod type squashfs (ro,nodev,relatime)
/dev/md2 on /usr/share/ca-certificates type ext4 (rw,relatime)
udev on /var/lib/snapd/hostfs/dev type devtmpfs (rw,nosuid,relatime,size=32912512k,nr_inodes=8228128,mode=755)
proc on /var/lib/snapd/hostfs/proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /var/lib/snapd/hostfs/sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /var/snap/lxd/common/shmounts/instances type tmpfs (rw,relatime,size=100k,mode=711)
tmpfs on /run/user/998 type tmpfs (rw,nosuid,nodev,relatime,size=6585864k,mode=700,uid=998,gid=998)
tmpfs on /var/lib/snapd/hostfs/run/user/998 type tmpfs (rw,nosuid,nodev,relatime,size=6585864k,mode=700,uid=998,gid=998)
lxcfs on /var/snap/lxd/common/shmounts/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
tmpfs on /etc type tmpfs (rw,relatime,mode=755)
tmpfs on /usr/share/misc type tmpfs (rw,relatime,mode=755)
/dev/md2 on /boot type ext4 (rw,relatime)

tomp · August 12, 2021, 1:30pm

Can you re submit that command output as text please, its tricky to see/search for something particular as you’re attaching them as images.

khalid_mrabti · August 12, 2021, 1:34pm

here is the file :
i couldnt upload the file in the reply, thats why i used google drive it:

the output:

/dev/md2 on /var/lib/snapd/hostfs type ext4 (rw,relatime)
tmpfs on /var/lib/snapd/hostfs/run type tmpfs (rw,nosuid,noexec,relatime,size=6585868k,mode=755)
tmpfs on /var/lib/snapd/hostfs/run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /var/lib/snapd/hostfs/run/snapd/ns type tmpfs (rw,nosuid,noexec,relatime,size=6585868k,mode=755)
tmpfs on /var/lib/snapd/hostfs/run/user/110 type tmpfs (rw,nosuid,nodev,relatime,size=6585864k,mode=700,uid=110,gid=117)
/dev/sda1 on /var/lib/snapd/hostfs/boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
/dev/md4 on /var/lib/snapd/hostfs/opt type ext4 (rw,relatime)
/var/lib/snapd/snaps/core20_1026.snap on /var/lib/snapd/hostfs/snap/core20/1026 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core18_2074.snap on /var/lib/snapd/hostfs/snap/core18/2074 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core_11316.snap on /var/lib/snapd/hostfs/snap/core/11316 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/certbot_1280.snap on /var/lib/snapd/hostfs/snap/certbot/1280 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/lxd_21039.snap on /var/lib/snapd/hostfs/snap/lxd/21039 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core20_1081.snap on /var/lib/snapd/hostfs/snap/core20/1081 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core_11420.snap on /var/lib/snapd/hostfs/snap/core/11420 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/certbot_1343.snap on /var/lib/snapd/hostfs/snap/certbot/1343 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/lxd_21260.snap on /var/lib/snapd/hostfs/snap/lxd/21260 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core18_2128.snap on /var/lib/snapd/hostfs/snap/core18/2128 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core20_1081.snap on / type squashfs (ro,nodev,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=32912512k,nr_inodes=8228128,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
/dev/md2 on /etc type ext4 (rw,relatime)
/dev/md2 on /home type ext4 (rw,relatime)
/dev/md2 on /root type ext4 (rw,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11231)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset,clone_children)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/md2 on /tmp type ext4 (rw,relatime)
/dev/md2 on /var/snap type ext4 (rw,relatime)
/dev/md2 on /var/lib/snapd type ext4 (rw,relatime)
/dev/md2 on /var/tmp type ext4 (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=6585868k,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,noexec,relatime,size=6585868k,mode=755)
tmpfs on /run/user/110 type tmpfs (rw,nosuid,nodev,relatime,size=6585864k,mode=700,uid=110,gid=117)
/dev/md2 on /usr/lib/modules type ext4 (rw,relatime)
/dev/md2 on /usr/lib/firmware type ext4 (rw,relatime)
/dev/md2 on /usr/src type ext4 (rw,relatime)
/dev/md2 on /var/log type ext4 (rw,relatime)
/dev/md2 on /media type ext4 (rw,relatime)
tmpfs on /run/netns type tmpfs (rw,nosuid,noexec,relatime,size=6585868k,mode=755)
/dev/md2 on /mnt type ext4 (rw,relatime)
/var/lib/snapd/snaps/core20_1081.snap on /etc/nsswitch.conf type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core20_1081.snap on /etc/apparmor type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core20_1081.snap on /etc/apparmor.d type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core_11420.snap on /usr/lib/snapd type squashfs (ro,nodev,relatime)
/dev/md2 on /snap type ext4 (rw,relatime)
/var/lib/snapd/snaps/core20_1026.snap on /snap/core20/1026 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core18_2074.snap on /snap/core18/2074 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core_11316.snap on /snap/core/11316 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/certbot_1280.snap on /snap/certbot/1280 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/lxd_21039.snap on /snap/lxd/21039 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core20_1081.snap on /snap/core20/1081 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core_11420.snap on /snap/core/11420 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/certbot_1343.snap on /snap/certbot/1343 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/lxd_21260.snap on /snap/lxd/21260 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core18_2128.snap on /snap/core18/2128 type squashfs (ro,nodev,relatime)
/dev/md2 on /var/lib/snapd/hostfs type ext4 (rw,relatime)
/dev/md2 on /tmp type ext4 (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/ptmx type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
tmpfs on /var/snap/lxd/common/lxd/devlxd type tmpfs (rw,relatime,size=100k,mode=755)
/dev/md2 on /var/snap/lxd/common/lxd/storage-pools type ext4 (rw,relatime)
/dev/md2 on /var/snap/lxd/common/lxd/devices type ext4 (rw,relatime)
tmpfs on /var/snap/lxd/common/shmounts type tmpfs (rw,relatime,size=1024k,mode=711)
/var/lib/snapd/snaps/lxd_21260.snap on /usr/bin/kmod type squashfs (ro,nodev,relatime)
/dev/md2 on /usr/share/ca-certificates type ext4 (rw,relatime)
udev on /var/lib/snapd/hostfs/dev type devtmpfs (rw,nosuid,relatime,size=32912512k,nr_inodes=8228128,mode=755)
proc on /var/lib/snapd/hostfs/proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /var/lib/snapd/hostfs/sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /var/snap/lxd/common/shmounts/instances type tmpfs (rw,relatime,size=100k,mode=711)
tmpfs on /run/user/998 type tmpfs (rw,nosuid,nodev,relatime,size=6585864k,mode=700,uid=998,gid=998)
tmpfs on /var/lib/snapd/hostfs/run/user/998 type tmpfs (rw,nosuid,nodev,relatime,size=6585864k,mode=700,uid=998,gid=998)
lxcfs on /var/snap/lxd/common/shmounts/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/md2 on /boot type ext4 (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
tmpfs on /etc type tmpfs (rw,relatime,mode=755)
tmpfs on /usr/share/misc type tmpfs (rw,relatime,mode=755)

tomp · August 12, 2021, 1:35pm

Thanks. BTW you can literally just paste it into the forum post wrapped in backticks for code view.

Seems like the simplest approach to keep it altogether

tomp · August 12, 2021, 1:43pm

Looks fine - no mounts for that pool I can see.

Does /dev/loop2 exist?

Also, does /var/snap/lxd/common/lxd/disks/default.img exist?

khalid_mrabti · August 12, 2021, 1:44pm

yes both are exists:
# ls -la /dev/loop2
brw-rw---- 1 root disk 7, 2 août 12 15:43 /dev/loop2
# ls -la /var/snap/lxd/common/lxd/disks/default.img
-rw------- 1 root root 322122547200 août 12 15:43 /var/snap/lxd/common/lxd/disks/default.img

tomp · August 12, 2021, 1:51pm

Are the instances running using the default storage pool?

If so, I suspect that this is the issue, they are holding an old reference to a still mounted storage pool, and somehow that storage pool has become unmounted inside the snap, yet still in use by the instances.

A reboot would most likely resolve this.

Any ideas @stgraber

tomp · August 12, 2021, 1:52pm

Whats the output of sudo losetup -a btw?

khalid_mrabti · August 12, 2021, 1:53pm

there is some containers using the default storage pool (default), and others uses a pool called rdv-storage-pool

khalid_mrabti · August 12, 2021, 1:54pm

# losetup -a

/dev/loop1: [2306]:15995746 (/var/lib/snapd/snaps/core_11316.snap)
/dev/loop19: [2306]:15995725 (/var/lib/snapd/snaps/lxd_20750.snap (deleted))
/dev/loop15: [2306]:15996544 (/var/lib/snapd/snaps/core18_2128.snap)
/dev/loop6: [2306]:15993222 (/lxd/common/lxd/disks/default.img)
/dev/loop13: [2306]:15996003 (/var/lib/snapd/snaps/core_11187.snap (deleted))
/dev/loop4: [2306]:15993006 (/var/lib/snapd/snaps/core_11420.snap)
/dev/loop11: [2306]:15996022 (/var/lib/snapd/snaps/core20_1081.snap)
/dev/loop2: [2306]:15993222 (/var/snap/lxd/common/lxd/disks/default.img)
/dev/loop0: [2306]:15993168 (/var/snap/lxd/common/lxd/disks/rdv-storage-pool.img)
/dev/loop9: [2306]:15995736 (/var/lib/snapd/snaps/core18_2074.snap)
/dev/loop7: [2306]:15996035 (/var/lib/snapd/snaps/lxd_20987.snap (deleted))
/dev/loop14: [2306]:15996016 (/var/lib/snapd/snaps/lxd_21260.snap)
/dev/loop5: [2306]:15995697 (/var/lib/snapd/snaps/lxd_21039.snap)
/dev/loop12: [2306]:15995745 (/var/lib/snapd/snaps/certbot_1343.snap)
/dev/loop3: [2306]:15996026 (/var/lib/snapd/snaps/certbot_1280.snap)
/dev/loop20: [2306]:15995424 (/var/lib/snapd/snaps/lxd_20806.snap (deleted))
/dev/loop10: [2306]:15995704 (/var/lib/snapd/snaps/core20_1026.snap)