Failed to start LXD vm because of bind disk

cola16 · October 19, 2022, 7:39am

Environment
HOST OS: Alma Linux 9
qemu:6.2.0
LXD version: 5.6
architecture: aarch64
LXD vm OS: ubuntu/22.04

when I tried to start ubuntu-vm, I got a error message.
Error: Failed to start device “USERDATA”: remove /var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.–USERDATA: device or resource busy

I can start the machine after run this commands:
rm -r /var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA

The ‘USERDATA device’ is a bind disk device:
lxc config device add ubuntu-vm USERDATA disk source=/-USERDATA path=/-USERDATA recursive=true propagation=rshared

When lxd vm is turned on, there are about 3 folders and files in /var/snap/lxd/common/lxd/devices/ubuntu-vm folder.
However, if I shutdown the lxd vm, all folders and files will be deleted except the directory.
It seems that this folder should be also deleted when lxd vm is stopped.
But lxd can not erase it and I can not start.

Is there a solution?

tomp · October 20, 2022, 10:11am

What does sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- findmnt show?

cola16 · October 20, 2022, 1:31pm

Thanky for your reply

here’s the result

TARGET                                                               SOURCE                                   FSTYPE      OPTIONS
/                                                                    /dev/loop3                               squashfs    ro,nodev,relatime,context=system_u:object_r:s
|-/dev                                                               devtmpfs                                 devtmpfs    rw,nosuid,seclabel,size=1906580k,nr_inodes=47
| |-/dev/ptmx                                                        devpts[/ptmx]                            devpts      rw,relatime,seclabel,gid=5,mode=620,ptmxmode=
| |-/dev/shm                                                         tmpfs                                    tmpfs       rw,nosuid,nodev,seclabel,inode64
| |-/dev/pts                                                         devpts                                   devpts      rw,nosuid,noexec,relatime,seclabel,gid=5,mode
| | `-/dev/pts                                                       devpts                                   devpts      rw,relatime,seclabel,gid=5,mode=620,ptmxmode=
| |-/dev/hugepages                                                   hugetlbfs                                hugetlbfs   rw,relatime,seclabel,pagesize=2M
| `-/dev/mqueue                                                      mqueue                                   mqueue      rw,nosuid,nodev,noexec,relatime,seclabel
|-/etc                                                               rpool/redhat/root[/etc]                  zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| |-/etc/nsswitch.conf                                               /dev/loop3[/etc/nsswitch.conf]           squashfs    ro,nodev,relatime,context=system_u:object_r:s
| |-/etc/ssl                                                         /dev/loop3[/etc/ssl]                     squashfs    ro,nodev,relatime,context=system_u:object_r:s
| `-/etc                                                             tmpfs                                    tmpfs       rw,relatime,seclabel,mode=755,inode64
|-/home                                                              rpool/redhat/root/home                   zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|-/root                                                              rpool/redhat/root/root                   zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|-/proc                                                              proc                                     proc        rw,nosuid,nodev,noexec,relatime
| |-/proc/sys/fs/binfmt_misc                                         systemd-1                                autofs      rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5
| | `-/proc/sys/fs/binfmt_misc                                       binfmt_misc                              binfmt_misc rw,nosuid,nodev,noexec,relatime
| `-/proc/fs/nfsd                                                    nfsd                                     nfsd        rw,relatime
|-/sys                                                               sysfs                                    sysfs       rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/kernel/security                                             securityfs                               securityfs  rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/cgroup                                                   cgroup2                                  cgroup2     rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/fs/pstore                                                   pstore                                   pstore      rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/firmware/efi/efivars                                        efivarfs                                 efivarfs    rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/bpf                                                      none                                     bpf         rw,nosuid,nodev,noexec,relatime,mode=700
| |-/sys/fs/selinux                                                  selinuxfs                                selinuxfs   rw,nosuid,noexec,relatime
| |-/sys/kernel/debug                                                debugfs                                  debugfs     rw,nosuid,nodev,noexec,relatime,seclabel
| | `-/sys/kernel/debug/tracing                                      tracefs                                  tracefs     rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/kernel/tracing                                              tracefs                                  tracefs     rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/fs/fuse/connections                                         fusectl                                  fusectl     rw,nosuid,nodev,noexec,relatime
| `-/sys/kernel/config                                               configfs                                 configfs    rw,nosuid,nodev,noexec,relatime
|-/tmp                                                               rpool/redhat/root[/tmp]                  zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| `-/tmp                                                             rpool/redhat/root[/tmp/snap.lxd/tmp]     zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|-/var/snap                                                          rpool/redhat/root/var/snap               zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| |-/var/snap/lxd/common/lxd/storage-pools                           rpool/redhat/root/var/snap[/lxd/common/lxd/storage-pools]
| |                                                                                                           zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| | `-/var/snap/lxd/common/lxd/storage-pools/lxd_pool/containers/lxd-dashboard
| |                                                                  user-pool/-USERDATA/littleSnap/lxd/containers/lxd-dashboard
| |                                                                                                           zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| |-/var/snap/lxd/common/lxd/devices                                 rpool/redhat/root/var/snap[/lxd/common/lxd/devices]
| |                                                                                                           zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| | `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA
| |                                                                  user-pool/-USERDATA                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| |   |-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/noSnap
| |   |                                                              user-pool/-USERDATA/noSnap               zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| |   | `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/noSnap/libvirt/swapStoragePool
| |   |                                                              user-pool/-USERDATA/noSnap/libvirt/swapStoragePool
| |   |                                                                                                       zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| |   |-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/littleSnap
| |   |                                                              user-pool/-USERDATA/littleSnap           zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| |   | `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/littleSnap/libvirt
| |   |                                                              user-pool/-USERDATA/littleSnap/libvirt   zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| |   |   |-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/littleSnap/libvirt/iso-pool
| |   |   |                                                          user-pool/-USERDATA/littleSnap/libvirt/iso-pool
| |   |   |                                                                                                   zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| |   |   `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/littleSnap/libvirt/storagePool
| |   |                                                              user-pool/-USERDATA/littleSnap/libvirt/storagePool
| |   |                                                                                                       zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| |   |     `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/littleSnap/libvirt/storagePool/default
| |   |                                                              user-pool/-USERDATA/littleSnap/libvirt/storagePool/default
| |   |                                                                                                       zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| |   `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/muchSnap
| |                                                                  user-pool/-USERDATA/muchSnap             zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| |     |-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/muchSnap/docker
| |     |                                                            user-pool/-USERDATA/muchSnap/docker      zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| |     `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/muchSnap/resources
| |                                                                  user-pool/-USERDATA/muchSnap/resources   zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| |-/var/snap/lxd/common/ns                                          tmpfs                                    tmpfs       rw,relatime,seclabel,size=1024k,mode=700,inod
| |-/var/snap/lxd/common/shmounts                                    tmpfs                                    tmpfs       rw,relatime,seclabel,size=1024k,mode=711,inod
| | |-/var/snap/lxd/common/shmounts/lxcfs                            lxcfs                                    fuse.lxcfs  rw,nosuid,nodev,relatime,user_id=0,group_id=0
| | `-/var/snap/lxd/common/shmounts/instances                        tmpfs                                    tmpfs       rw,relatime,seclabel,size=100k,mode=711,inode
| `-/var/snap/lxd/common/lxd/devlxd                                  tmpfs                                    tmpfs       rw,relatime,seclabel,size=100k,mode=755,inode
|-/var/lib/snapd                                                     rpool/redhat/root/var/lib/snapd          zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| |-/var/lib/snapd/snap/snapd/17339                                  /dev/loop0                               squashfs    ro,nodev,relatime,context=system_u:object_r:s
| |-/var/lib/snapd/snap/lxd/23687                                    /dev/loop1                               squashfs    ro,nodev,relatime,context=system_u:object_r:s
| |-/var/lib/snapd/snap/snapd/17032                                  /dev/loop2                               squashfs    ro,nodev,relatime,context=system_u:object_r:s
| |-/var/lib/snapd/snap/core20/1627                                  /dev/loop3                               squashfs    ro,nodev,relatime,context=system_u:object_r:s
| `-/var/lib/snapd/hostfs                                            rpool/redhat/root/var/lib/snapd[/hostfs] zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|   `-/var/lib/snapd/hostfs                                          rpool/redhat/root                        zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     |-/var/lib/snapd/hostfs/run                                    tmpfs                                    tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inod
|     | |-/var/lib/snapd/hostfs/run/user/4261                        tmpfs                                    tmpfs       rw,nosuid,nodev,relatime,seclabel,size=384948
|     | `-/var/lib/snapd/hostfs/run/snapd/ns                         tmpfs[/snapd/ns]                         tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inod
|     |-/var/lib/snapd/hostfs/boot/efis/scsi-33001237923792379-part2 /dev/sda2                                vfat        rw,relatime,fmask=0022,dmask=0022,codepage=43
|     |-/var/lib/snapd/hostfs/boot                                   bpool/redhat/boot                        zfs         rw,nodev,relatime,seclabel,xattr,posixacl,cas
|     |-/var/lib/snapd/hostfs/-USERDATA                              user-pool/-USERDATA                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     | |-/var/lib/snapd/hostfs/-USERDATA/noSnap                     user-pool/-USERDATA/noSnap               zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     | | `-/var/lib/snapd/hostfs/-USERDATA/noSnap/libvirt/swapStoragePool
|     | |                                                            user-pool/-USERDATA/noSnap/libvirt/swapStoragePool
|     | |                                                                                                     zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     | |-/var/lib/snapd/hostfs/-USERDATA/littleSnap                 user-pool/-USERDATA/littleSnap           zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     | | `-/var/lib/snapd/hostfs/-USERDATA/littleSnap/libvirt       user-pool/-USERDATA/littleSnap/libvirt   zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     | |   |-/var/lib/snapd/hostfs/-USERDATA/littleSnap/libvirt/iso-pool
|     | |   |                                                        user-pool/-USERDATA/littleSnap/libvirt/iso-pool
|     | |   |                                                                                                 zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     | |   `-/var/lib/snapd/hostfs/-USERDATA/littleSnap/libvirt/storagePool
|     | |                                                            user-pool/-USERDATA/littleSnap/libvirt/storagePool
|     | |                                                                                                     zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     | |     `-/var/lib/snapd/hostfs/-USERDATA/littleSnap/libvirt/storagePool/default
|     | |                                                            user-pool/-USERDATA/littleSnap/libvirt/storagePool/default
|     | |                                                                                                     zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     | `-/var/lib/snapd/hostfs/-USERDATA/muchSnap                   user-pool/-USERDATA/muchSnap             zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     |   |-/var/lib/snapd/hostfs/-USERDATA/muchSnap/docker          user-pool/-USERDATA/muchSnap/docker      zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     |   `-/var/lib/snapd/hostfs/-USERDATA/muchSnap/resources       user-pool/-USERDATA/muchSnap/resources   zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     |-/var/lib/snapd/hostfs/root                                   rpool/redhat/root/root                   zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     |-/var/lib/snapd/hostfs/home                                   rpool/redhat/root/home                   zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     |-/var/lib/snapd/hostfs/var/lib                                rpool/redhat/root/var/lib                zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     | |-/var/lib/snapd/hostfs/var/lib/NetworkManager               rpool/redhat/root/var/lib/NetworkManager zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     | |-/var/lib/snapd/hostfs/var/lib/nfs                          rpool/redhat/NoSnap-var/lib/nfs          zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     | | `-/var/lib/snapd/hostfs/var/lib/nfs/rpc_pipefs             sunrpc                                   rpc_pipefs  rw,relatime
|     | |-/var/lib/snapd/hostfs/var/lib/docker                       rpool/redhat/root/var/lib/docker         zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     | `-/var/lib/snapd/hostfs/var/lib/snapd                        rpool/redhat/root/var/lib/snapd          zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     |   |-/var/lib/snapd/hostfs/var/lib/snapd/snap/snapd/17339     /dev/loop0                               squashfs    ro,nodev,relatime,context=system_u:object_r:s
|     |   |-/var/lib/snapd/hostfs/var/lib/snapd/snap/lxd/23687       /dev/loop1                               squashfs    ro,nodev,relatime,context=system_u:object_r:s
|     |   |-/var/lib/snapd/hostfs/var/lib/snapd/snap/snapd/17032     /dev/loop2                               squashfs    ro,nodev,relatime,context=system_u:object_r:s
|     |   `-/var/lib/snapd/hostfs/var/lib/snapd/snap/core20/1627     /dev/loop3                               squashfs    ro,nodev,relatime,context=system_u:object_r:s
|     |-/var/lib/snapd/hostfs/var/tmp                                rpool/redhat/NoSnap-var/tmp              zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     |-/var/lib/snapd/hostfs/usr/local                              rpool/redhat/root/usr/local              zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     |-/var/lib/snapd/hostfs/var/cache                              rpool/redhat/NoSnap-var/cache            zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     |-/var/lib/snapd/hostfs/var/spool                              rpool/redhat/root/var/spool              zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     |-/var/lib/snapd/hostfs/var/log                                rpool/redhat/root/var/log                zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     |-/var/lib/snapd/hostfs/var/snap                               rpool/redhat/root/var/snap               zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|     | `-/var/lib/snapd/hostfs/var/snap/lxd/common/ns               tmpfs                                    tmpfs       rw,relatime,seclabel,size=1024k,mode=700,inod
|     |-/var/lib/snapd/hostfs/dev                                    devtmpfs                                 devtmpfs    rw,nosuid,seclabel,size=1906580k,nr_inodes=47
|     |-/var/lib/snapd/hostfs/proc                                   proc                                     proc        rw,nosuid,nodev,noexec,relatime
|     `-/var/lib/snapd/hostfs/sys                                    sysfs                                    sysfs       rw,nosuid,nodev,noexec,relatime,seclabel
|-/var/tmp                                                           rpool/redhat/NoSnap-var/tmp              zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|-/run                                                               tmpfs                                    tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inod
| |-/run/user/4261                                                   tmpfs                                    tmpfs       rw,nosuid,nodev,relatime,seclabel,size=384948
| |-/run/snapd/ns                                                    tmpfs[/snapd/ns]                         tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inod
| |-/run/media                                                       tmpfs[/media]                            tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inod
| |-/run/netns                                                       tmpfs[/netns]                            tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inod
| `-/run                                                             tmpfs                                    tmpfs       rw,nosuid,nodev,relatime,seclabel,mode=755,in
|-/usr/lib/modules                                                   rpool/redhat/root[/usr/lib/modules]      zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|-/usr/lib/firmware                                                  rpool/redhat/root[/usr/lib/firmware]     zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|-/usr/src                                                           rpool/redhat/root[/usr/src]              zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|-/var/log                                                           rpool/redhat/root/var/log                zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|-/media                                                             tmpfs[/media]                            tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inod
|-/mnt                                                               rpool/redhat/root[/mnt]                  zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|-/usr/lib/snapd                                                     rpool/redhat/root[/usr/libexec/snapd]    zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
|-/snap                                                              rpool/redhat/root/var/lib/snapd[/snap]   zfs         rw,relatime,seclabel,xattr,posixacl,casesensi
| |-/snap/snapd/17339                                                /dev/loop0                               squashfs    ro,nodev,relatime,context=system_u:object_r:s
| |-/snap/lxd/23687                                                  /dev/loop1                               squashfs    ro,nodev,relatime,context=system_u:object_r:s
| |-/snap/snapd/17032                                                /dev/loop2                               squashfs    ro,nodev,relatime,context=system_u:object_r:s
| `-/snap/core20/1627                                                /dev/loop3                               squashfs    ro,nodev,relatime,context=system_u:object_r:s
|-/usr/bin/kmod                                                      /dev/loop1[/wrappers/kmod]               squashfs    ro,nodev,relatime,context=system_u:object_r:s
|-/boot                                                              bpool/redhat/boot                        zfs         rw,nodev,relatime,seclabel,xattr,posixacl,cas
`-/usr/share/misc                                                    tmpfs                                    tmpfs       rw,relatime,seclabel,mode=755,inode64

tomp · October 20, 2022, 1:52pm

Please can you show lxc config show <instance> --expanded and sudo zfs list

Also I don’t quite follow you about removing the /var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA does it fail to get removed after its recreated on next start up?

cola16 · October 21, 2022, 12:59am

# after I remove /var/snap/lxd/common/lxd/devices/ubuntu-vm/, the machine works
╰─ lxc start ubuntu-vm
#there are some directories
╰─ sudo ls -al /var/snap/lxd/common/lxd/devices/ubuntu-vm/
total 17
drwx--x--x. 4 root root   5 Oct 21 10:01 .
drwxrwxrwx. 5 root root   5 Oct 20 22:08 ..
drwx------. 2 root root   2 Oct 21 10:01 config.mount
drwx------. 2 root root   2 Oct 21 10:01 disk.USERDATA.--USERDATA
-rw-r--r--. 1 root root 229 Oct 21 10:01 USERDATA.pid

╰─ lxc stop ubuntu-vm
# after stop the machine, the only  directory left
╰─ sudo ls -al /var/snap/lxd/common/lxd/devices/ubuntu-vm/
total 3
drwx--x--x. 3 root root 3 Oct 21 10:02 .
drwxrwxrwx. 5 root root 5 Oct 20 22:08 ..
drwx------. 2 root root 2 Oct 21 10:01 disk.USERDATA.--USERDATA

# can not start
╰─ lxc start ubuntu-vm
Error: Failed to start device "USERDATA": remove /var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA: device or resource busy
Try `lxc info --show-log ubuntu-vm` for more info

╰─ sudo rm -r /var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA

╰─ lxc start ubuntu-vm 
╰─

When I boot the vm, some directories are created in /var/snap/lxd/common/lxd/devices/ubuntu-vm/. after I shutdown the vm, Other directories are removed in /var/snap/lxd/common/lxd/devices/ubuntu-vm/.
But disk.USERDATA.--USERDATA doesn’t remved by shutdown the machine.

I can remove it.
I can boot the machine after remove the dir by me

the machine doesn’t boot until removed by me

╰─ lxc config show ubuntu-vm --expanded
architecture: aarch64
config:
  image.architecture: arm64
  image.description: Ubuntu jammy arm64 (20221014_07:43)
  image.os: Ubuntu
  image.release: jammy
  image.serial: "20221014_07:43"
  image.type: disk-kvm.img
  image.variant: default
  security.secureboot: "false"
  volatile.base_image: 61fe259f5411ed27962fe5dc2a7bf96e7991766d4a447933c54d327e267cdbff
  volatile.cloud-init.instance-id: 93eebb43-e11e-42a7-9ac9-32df8edce9b5
  volatile.eth0.hwaddr: 00:16:3e:54:ce:06
  volatile.last_state.power: STOPPED
  volatile.last_state.ready: "false"
  volatile.uuid: db288f4a-681b-4abd-92a9-b5530be93fe6
  volatile.vsock_id: "15"
devices:
  USERDATA:
    path: /-USERDATA
    propagation: rshared
    recursive: "true"
    source: /-USERDATA
    type: disk
  eth0:
    name: eth0
    nictype: bridged
    parent: br-local
    type: nic
  root:
    path: /
    pool: lxd_pool
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

╰─ sudo zfs list                                                                                                                                                                                                                                                                                                           ─╯
NAME                                                                                                                 USED  AVAIL  REFER  MOUNTPOINT
bpool                                                                                                                104M  3.52G    96K  /boot
bpool/redhat                                                                                                         103M  3.52G    96K  none
bpool/redhat/boot                                                                                                    103M  3.52G   103M  /boot
rpool                                                                                                               12.4G  45.3G    96K  /
rpool/redhat                                                                                                        12.3G  45.3G    96K  none
rpool/redhat/NoSnap-var                                                                                              237M  45.3G    96K  /var
rpool/redhat/NoSnap-var/cache                                                                                        158M  45.3G   158M  /var/cache
rpool/redhat/NoSnap-var/lib                                                                                          208K  45.3G    96K  /var/lib
rpool/redhat/NoSnap-var/lib/nfs                                                                                      112K  45.3G   112K  /var/lib/nfs
rpool/redhat/NoSnap-var/tmp                                                                                         78.5M  45.3G  78.5M  /var/tmp
rpool/redhat/root                                                                                                   12.0G  45.3G  1.93G  /
rpool/redhat/root/home                                                                                               220M  45.3G   202M  /home
rpool/redhat/root/root                                                                                              1.30M  45.3G   768K  /root
rpool/redhat/root/usr                                                                                               5.12M  45.3G    96K  /usr
rpool/redhat/root/usr/local                                                                                         5.03M  45.3G  2.62M  /usr/local
rpool/redhat/root/var                                                                                               9.69G  45.3G    96K  /var
rpool/redhat/root/var/lib                                                                                            858M  45.3G   344M  /var/lib
rpool/redhat/root/var/lib/NetworkManager                                                                            8.74M  45.3G   256K  /var/lib/NetworkManager
rpool/redhat/root/var/lib/docker                                                                                      96K  45.3G    96K  /var/lib/docker
rpool/redhat/root/var/lib/snapd                                                                                      385M  45.3G   383M  /var/lib/snapd
rpool/redhat/root/var/log                                                                                           78.5M  45.3G  27.2M  /var/log
rpool/redhat/root/var/snap                                                                                          8.78G  45.3G  5.96G  /var/snap
rpool/redhat/root/var/spool                                                                                         2.19M  45.3G   128K  /var/spool
user-pool                                                                                                           48.1G   328G    96K  /
user-pool/-USERDATA                                                                                                 47.8G   328G  66.3M  /-USERDATA
user-pool/-USERDATA/littleSnap                                                                                      40.8G   328G    96K  /-USERDATA/littleSnap
user-pool/-USERDATA/littleSnap/libvirt                                                                              32.3G   328G    96K  /-USERDATA/littleSnap/libvirt
user-pool/-USERDATA/littleSnap/libvirt/iso-pool                                                                     6.55G   328G  6.55G  /-USERDATA/littleSnap/libvirt/iso-pool
user-pool/-USERDATA/littleSnap/libvirt/storagePool                                                                  25.7G   328G    96K  /-USERDATA/littleSnap/libvirt/storagePool
user-pool/-USERDATA/littleSnap/libvirt/storagePool/default                                                          25.7G   328G  4.99G  /-USERDATA/littleSnap/libvirt/storagePool/default
user-pool/-USERDATA/littleSnap/lxd                                                                                  8.54G   328G    96K  legacy
user-pool/-USERDATA/littleSnap/lxd/buckets                                                                            96K   328G    96K  legacy
user-pool/-USERDATA/littleSnap/lxd/containers                                                                       3.73G   328G    96K  legacy
user-pool/-USERDATA/littleSnap/lxd/containers/OL                                                                    3.54G   328G  3.60G  legacy
user-pool/-USERDATA/littleSnap/lxd/containers/lxd-dashboard                                                          190M   328G   362M  legacy
user-pool/-USERDATA/littleSnap/lxd/custom                                                                            200K   328G    96K  legacy
user-pool/-USERDATA/littleSnap/lxd/custom/default_test                                                               104K   328G    96K  legacy
user-pool/-USERDATA/littleSnap/lxd/deleted                                                                           253M   328G    96K  legacy
user-pool/-USERDATA/littleSnap/lxd/deleted/buckets                                                                    96K   328G    96K  legacy
user-pool/-USERDATA/littleSnap/lxd/deleted/containers                                                                 96K   328G    96K  legacy
user-pool/-USERDATA/littleSnap/lxd/deleted/custom                                                                     96K   328G    96K  legacy
user-pool/-USERDATA/littleSnap/lxd/deleted/images                                                                    252M   328G    96K  legacy
user-pool/-USERDATA/littleSnap/lxd/deleted/images/e0ea3ba070f63ec5f019e1cdd00784a4130426ab98ea5ca6e81de53e1be13468   252M   328G   252M  legacy
user-pool/-USERDATA/littleSnap/lxd/deleted/virtual-machines                                                           96K   328G    96K  legacy
user-pool/-USERDATA/littleSnap/lxd/images                                                                           4.56G   328G    96K  legacy
user-pool/-USERDATA/littleSnap/lxd/images/46dc39a9cab5844795e7a8c1cbcaf42b0095ab8e28c7e27a974c2aa56ec0d4ae           100K  99.9M   100K  legacy
user-pool/-USERDATA/littleSnap/lxd/images/46dc39a9cab5844795e7a8c1cbcaf42b0095ab8e28c7e27a974c2aa56ec0d4ae.block    3.67G   328G  3.67G  -
user-pool/-USERDATA/littleSnap/lxd/images/5582f7885f47195c6c813309022dfeedc1cb6867b4542cbbf21c556cd0e3339f           268M   328G   268M  legacy
user-pool/-USERDATA/littleSnap/lxd/images/61fe259f5411ed27962fe5dc2a7bf96e7991766d4a447933c54d327e267cdbff           112K  99.9M   104K  legacy
user-pool/-USERDATA/littleSnap/lxd/images/61fe259f5411ed27962fe5dc2a7bf96e7991766d4a447933c54d327e267cdbff.block     392M   328G   392M  -
user-pool/-USERDATA/littleSnap/lxd/images/f41351143c5ec014ecedeb7fad7b33e2a7edc3336d2f0153120d4b6455d4496b           252M   328G   252M  legacy
user-pool/-USERDATA/littleSnap/lxd/virtual-machines                                                                 9.19M   328G    96K  legacy
user-pool/-USERDATA/littleSnap/lxd/virtual-machines/ubuntu-vm                                                       5.76M  94.2M  5.80M  legacy
user-pool/-USERDATA/littleSnap/lxd/virtual-machines/ubuntu-vm.block                                                 3.34M   328G   393M  -
user-pool/-USERDATA/muchSnap                                                                                        2.92G   328G   104K  /-USERDATA/muchSnap
user-pool/-USERDATA/muchSnap/docker                                                                                 2.92G   328G  1.01G  /-USERDATA/muchSnap/docker
user-pool/-USERDATA/muchSnap/resources                                                                              4.51M   328G   172K  /-USERDATA/muchSnap/resources
user-pool/-USERDATA/noSnap                                                                                          4.03G   328G  3.33G  /-USERDATA/noSnap
user-pool/-USERDATA/noSnap/libvirt                                                                                   717M   328G    96K  /-USERDATA/noSnap/libvirt
user-pool/-USERDATA/noSnap/libvirt/swapStoragePool                                                                   717M   328G   717M  /-USERDATA/noSnap/libvirt/swapStoragePool

cola16 · October 21, 2022, 1:15am

Here’s more information.
after remove the directory manually, sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- findmnt result has changed.
/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA be disappeared.

#stop the vm
╰─ lxc stop ubuntu-vm
─ sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- findmnt
TARGET                                                                                                                   SOURCE                                                      FSTYPE      OPTIONS
/                                                                                                                        /dev/loop3                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|-/dev                                                                                                                   devtmpfs                                                    devtmpfs    rw,nosuid,seclabel,size=1906580k,nr_inodes=476645,mode=755,inode64
| |-/dev/ptmx                                                                                                            devpts[/ptmx]                                               devpts      rw,relatime,seclabel,gid=5,mode=620,ptmxmode=666
| |-/dev/shm                                                                                                             tmpfs                                                       tmpfs       rw,nosuid,nodev,seclabel,inode64
| |-/dev/pts                                                                                                             devpts                                                      devpts      rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000
| | `-/dev/pts                                                                                                           devpts                                                      devpts      rw,relatime,seclabel,gid=5,mode=620,ptmxmode=666
| |-/dev/hugepages                                                                                                       hugetlbfs                                                   hugetlbfs   rw,relatime,seclabel,pagesize=2M
| `-/dev/mqueue                                                                                                          mqueue                                                      mqueue      rw,nosuid,nodev,noexec,relatime,seclabel
|-/etc                                                                                                                   rpool/redhat/root[/etc]                                     zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/etc/nsswitch.conf                                                                                                   /dev/loop3[/etc/nsswitch.conf]                              squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/etc/ssl                                                                                                             /dev/loop3[/etc/ssl]                                        squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| `-/etc                                                                                                                 tmpfs                                                       tmpfs       rw,relatime,seclabel,mode=755,inode64
|-/home                                                                                                                  rpool/redhat/root/home                                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/root                                                                                                                  rpool/redhat/root/root                                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/proc                                                                                                                  proc                                                        proc        rw,nosuid,nodev,noexec,relatime
| |-/proc/sys/fs/binfmt_misc                                                                                             systemd-1                                                   autofs      rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=12903
| | `-/proc/sys/fs/binfmt_misc                                                                                           binfmt_misc                                                 binfmt_misc rw,nosuid,nodev,noexec,relatime
| `-/proc/fs/nfsd                                                                                                        nfsd                                                        nfsd        rw,relatime
|-/sys                                                                                                                   sysfs                                                       sysfs       rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/kernel/security                                                                                                 securityfs                                                  securityfs  rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/cgroup                                                                                                       cgroup2                                                     cgroup2     rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/fs/pstore                                                                                                       pstore                                                      pstore      rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/firmware/efi/efivars                                                                                            efivarfs                                                    efivarfs    rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/bpf                                                                                                          none                                                        bpf         rw,nosuid,nodev,noexec,relatime,mode=700
| |-/sys/fs/selinux                                                                                                      selinuxfs                                                   selinuxfs   rw,nosuid,noexec,relatime
| |-/sys/kernel/debug                                                                                                    debugfs                                                     debugfs     rw,nosuid,nodev,noexec,relatime,seclabel
| | `-/sys/kernel/debug/tracing                                                                                          tracefs                                                     tracefs     rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/kernel/tracing                                                                                                  tracefs                                                     tracefs     rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/fs/fuse/connections                                                                                             fusectl                                                     fusectl     rw,nosuid,nodev,noexec,relatime
| `-/sys/kernel/config                                                                                                   configfs                                                    configfs    rw,nosuid,nodev,noexec,relatime
|-/tmp                                                                                                                   rpool/redhat/root[/tmp]                                     zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| `-/tmp                                                                                                                 rpool/redhat/root[/tmp/snap.lxd/tmp]                        zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/var/snap                                                                                                              rpool/redhat/root/var/snap                                  zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/var/snap/lxd/common/lxd/storage-pools                                                                               rpool/redhat/root/var/snap[/lxd/common/lxd/storage-pools]   zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| | `-/var/snap/lxd/common/lxd/storage-pools/lxd_pool/containers/lxd-dashboard                                           user-pool/-USERDATA/littleSnap/lxd/containers/lxd-dashboard zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/var/snap/lxd/common/lxd/devices                                                                                     rpool/redhat/root/var/snap[/lxd/common/lxd/devices]         zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| | `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA                                                user-pool/-USERDATA                                         zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |   |-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/noSnap                                       user-pool/-USERDATA/noSnap                                  zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |   | `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/noSnap/libvirt/swapStoragePool             user-pool/-USERDATA/noSnap/libvirt/swapStoragePool          zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |   |-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/littleSnap                                   user-pool/-USERDATA/littleSnap                              zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |   | `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/littleSnap/libvirt                         user-pool/-USERDATA/littleSnap/libvirt                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |   |   |-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/littleSnap/libvirt/iso-pool              user-pool/-USERDATA/littleSnap/libvirt/iso-pool             zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |   |   `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/littleSnap/libvirt/storagePool           user-pool/-USERDATA/littleSnap/libvirt/storagePool          zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |   |     `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/littleSnap/libvirt/storagePool/default user-pool/-USERDATA/littleSnap/libvirt/storagePool/default  zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |   `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/muchSnap                                     user-pool/-USERDATA/muchSnap                                zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |     |-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/muchSnap/docker                            user-pool/-USERDATA/muchSnap/docker                         zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |     `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/muchSnap/resources                         user-pool/-USERDATA/muchSnap/resources                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/var/snap/lxd/common/ns                                                                                              tmpfs                                                       tmpfs       rw,relatime,seclabel,size=1024k,mode=700,inode64
| |-/var/snap/lxd/common/shmounts                                                                                        tmpfs                                                       tmpfs       rw,relatime,seclabel,size=1024k,mode=711,inode64
| | |-/var/snap/lxd/common/shmounts/lxcfs                                                                                lxcfs                                                       fuse.lxcfs  rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
| | `-/var/snap/lxd/common/shmounts/instances                                                                            tmpfs                                                       tmpfs       rw,relatime,seclabel,size=100k,mode=711,inode64
| `-/var/snap/lxd/common/lxd/devlxd                                                                                      tmpfs                                                       tmpfs       rw,relatime,seclabel,size=100k,mode=755,inode64
|-/var/lib/snapd                                                                                                         rpool/redhat/root/var/lib/snapd                             zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/var/lib/snapd/snap/snapd/17339                                                                                      /dev/loop0                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/var/lib/snapd/snap/lxd/23687                                                                                        /dev/loop1                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/var/lib/snapd/snap/snapd/17032                                                                                      /dev/loop2                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/var/lib/snapd/snap/core20/1627                                                                                      /dev/loop3                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| `-/var/lib/snapd/hostfs                                                                                                rpool/redhat/root/var/lib/snapd[/hostfs]                    zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|   `-/var/lib/snapd/hostfs                                                                                              rpool/redhat/root                                           zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/run                                                                                        tmpfs                                                       tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
|     | |-/var/lib/snapd/hostfs/run/user/4261                                                                            tmpfs                                                       tmpfs       rw,nosuid,nodev,relatime,seclabel,size=384948k,nr_inodes=96237,mode=700,uid=4261,gid=100,inode64
|     | `-/var/lib/snapd/hostfs/run/snapd/ns                                                                             tmpfs[/snapd/ns]                                            tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
|     |-/var/lib/snapd/hostfs/boot/efis/scsi-33001237923792379-part2                                                     /dev/sda2                                                   vfat        rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro
|     |-/var/lib/snapd/hostfs/boot                                                                                       bpool/redhat/boot                                           zfs         rw,nodev,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/-USERDATA                                                                                  user-pool/-USERDATA                                         zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |-/var/lib/snapd/hostfs/-USERDATA/noSnap                                                                         user-pool/-USERDATA/noSnap                                  zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | | `-/var/lib/snapd/hostfs/-USERDATA/noSnap/libvirt/swapStoragePool                                               user-pool/-USERDATA/noSnap/libvirt/swapStoragePool          zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |-/var/lib/snapd/hostfs/-USERDATA/littleSnap                                                                     user-pool/-USERDATA/littleSnap                              zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | | `-/var/lib/snapd/hostfs/-USERDATA/littleSnap/libvirt                                                           user-pool/-USERDATA/littleSnap/libvirt                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |   |-/var/lib/snapd/hostfs/-USERDATA/littleSnap/libvirt/iso-pool                                                user-pool/-USERDATA/littleSnap/libvirt/iso-pool             zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |   `-/var/lib/snapd/hostfs/-USERDATA/littleSnap/libvirt/storagePool                                             user-pool/-USERDATA/littleSnap/libvirt/storagePool          zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |     `-/var/lib/snapd/hostfs/-USERDATA/littleSnap/libvirt/storagePool/default                                   user-pool/-USERDATA/littleSnap/libvirt/storagePool/default  zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | `-/var/lib/snapd/hostfs/-USERDATA/muchSnap                                                                       user-pool/-USERDATA/muchSnap                                zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |   |-/var/lib/snapd/hostfs/-USERDATA/muchSnap/docker                                                              user-pool/-USERDATA/muchSnap/docker                         zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |   `-/var/lib/snapd/hostfs/-USERDATA/muchSnap/resources                                                           user-pool/-USERDATA/muchSnap/resources                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/root                                                                                       rpool/redhat/root/root                                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/home                                                                                       rpool/redhat/root/home                                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/var/lib                                                                                    rpool/redhat/root/var/lib                                   zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |-/var/lib/snapd/hostfs/var/lib/NetworkManager                                                                   rpool/redhat/root/var/lib/NetworkManager                    zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |-/var/lib/snapd/hostfs/var/lib/nfs                                                                              rpool/redhat/NoSnap-var/lib/nfs                             zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | | `-/var/lib/snapd/hostfs/var/lib/nfs/rpc_pipefs                                                                 sunrpc                                                      rpc_pipefs  rw,relatime
|     | |-/var/lib/snapd/hostfs/var/lib/docker                                                                           rpool/redhat/root/var/lib/docker                            zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | `-/var/lib/snapd/hostfs/var/lib/snapd                                                                            rpool/redhat/root/var/lib/snapd                             zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |   |-/var/lib/snapd/hostfs/var/lib/snapd/snap/snapd/17339                                                         /dev/loop0                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|     |   |-/var/lib/snapd/hostfs/var/lib/snapd/snap/lxd/23687                                                           /dev/loop1                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|     |   |-/var/lib/snapd/hostfs/var/lib/snapd/snap/snapd/17032                                                         /dev/loop2                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|     |   `-/var/lib/snapd/hostfs/var/lib/snapd/snap/core20/1627                                                         /dev/loop3                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|     |-/var/lib/snapd/hostfs/var/tmp                                                                                    rpool/redhat/NoSnap-var/tmp                                 zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/usr/local                                                                                  rpool/redhat/root/usr/local                                 zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/var/cache                                                                                  rpool/redhat/NoSnap-var/cache                               zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/var/spool                                                                                  rpool/redhat/root/var/spool                                 zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/var/log                                                                                    rpool/redhat/root/var/log                                   zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/var/snap                                                                                   rpool/redhat/root/var/snap                                  zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | `-/var/lib/snapd/hostfs/var/snap/lxd/common/ns                                                                   tmpfs                                                       tmpfs       rw,relatime,seclabel,size=1024k,mode=700,inode64
|     |-/var/lib/snapd/hostfs/dev                                                                                        devtmpfs                                                    devtmpfs    rw,nosuid,seclabel,size=1906580k,nr_inodes=476645,mode=755,inode64
|     |-/var/lib/snapd/hostfs/proc                                                                                       proc                                                        proc        rw,nosuid,nodev,noexec,relatime
|     `-/var/lib/snapd/hostfs/sys                                                                                        sysfs                                                       sysfs       rw,nosuid,nodev,noexec,relatime,seclabel
|-/var/tmp                                                                                                               rpool/redhat/NoSnap-var/tmp                                 zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/run                                                                                                                   tmpfs                                                       tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
| |-/run/user/4261                                                                                                       tmpfs                                                       tmpfs       rw,nosuid,nodev,relatime,seclabel,size=384948k,nr_inodes=96237,mode=700,uid=4261,gid=100,inode64
| |-/run/snapd/ns                                                                                                        tmpfs[/snapd/ns]                                            tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
| |-/run/media                                                                                                           tmpfs[/media]                                               tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
| |-/run/netns                                                                                                           tmpfs[/netns]                                               tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
| `-/run                                                                                                                 tmpfs                                                       tmpfs       rw,nosuid,nodev,relatime,seclabel,mode=755,inode64
|-/usr/lib/modules                                                                                                       rpool/redhat/root[/usr/lib/modules]                         zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/usr/lib/firmware                                                                                                      rpool/redhat/root[/usr/lib/firmware]                        zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/usr/src                                                                                                               rpool/redhat/root[/usr/src]                                 zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/var/log                                                                                                               rpool/redhat/root/var/log                                   zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/media                                                                                                                 tmpfs[/media]                                               tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
|-/mnt                                                                                                                   rpool/redhat/root[/mnt]                                     zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/usr/lib/snapd                                                                                                         rpool/redhat/root[/usr/libexec/snapd]                       zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/snap                                                                                                                  rpool/redhat/root/var/lib/snapd[/snap]                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/snap/snapd/17339                                                                                                    /dev/loop0                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/snap/lxd/23687                                                                                                      /dev/loop1                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/snap/snapd/17032                                                                                                    /dev/loop2                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| `-/snap/core20/1627                                                                                                    /dev/loop3                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|-/usr/bin/kmod                                                                                                          /dev/loop1[/wrappers/kmod]                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|-/boot                                                                                                                  bpool/redhat/boot                                           zfs         rw,nodev,relatime,seclabel,xattr,posixacl,casesensitive
`-/usr/share/misc                                                                                                        tmpfs                                                       tmpfs       rw,relatime,seclabel,mode=755,inode64
╭─    ~ ▓▒░────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────░▒▓ ✔ │ with host.rscola16@zfs  at 10:09:10  ─╮
#remove the dir
╰─ sudo rm -r /var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA                                                                                                                                                                                                                                          ─╯
╭─    ~ ▓▒░────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────░▒▓ ✔ │ with host.rscola16@zfs  at 10:09:58  ─╮
╰─ sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- findmnt                                                                                                                                                                                                                                                                   ─╯
TARGET                                                                                 SOURCE                                                      FSTYPE      OPTIONS
/                                                                                      /dev/loop3                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|-/dev                                                                                 devtmpfs                                                    devtmpfs    rw,nosuid,seclabel,size=1906580k,nr_inodes=476645,mode=755,inode64
| |-/dev/ptmx                                                                          devpts[/ptmx]                                               devpts      rw,relatime,seclabel,gid=5,mode=620,ptmxmode=666
| |-/dev/shm                                                                           tmpfs                                                       tmpfs       rw,nosuid,nodev,seclabel,inode64
| |-/dev/pts                                                                           devpts                                                      devpts      rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000
| | `-/dev/pts                                                                         devpts                                                      devpts      rw,relatime,seclabel,gid=5,mode=620,ptmxmode=666
| |-/dev/hugepages                                                                     hugetlbfs                                                   hugetlbfs   rw,relatime,seclabel,pagesize=2M
| `-/dev/mqueue                                                                        mqueue                                                      mqueue      rw,nosuid,nodev,noexec,relatime,seclabel
|-/etc                                                                                 rpool/redhat/root[/etc]                                     zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/etc/nsswitch.conf                                                                 /dev/loop3[/etc/nsswitch.conf]                              squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/etc/ssl                                                                           /dev/loop3[/etc/ssl]                                        squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| `-/etc                                                                               tmpfs                                                       tmpfs       rw,relatime,seclabel,mode=755,inode64
|-/home                                                                                rpool/redhat/root/home                                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/root                                                                                rpool/redhat/root/root                                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/proc                                                                                proc                                                        proc        rw,nosuid,nodev,noexec,relatime
| |-/proc/sys/fs/binfmt_misc                                                           systemd-1                                                   autofs      rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=12903
| | `-/proc/sys/fs/binfmt_misc                                                         binfmt_misc                                                 binfmt_misc rw,nosuid,nodev,noexec,relatime
| `-/proc/fs/nfsd                                                                      nfsd                                                        nfsd        rw,relatime
|-/sys                                                                                 sysfs                                                       sysfs       rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/kernel/security                                                               securityfs                                                  securityfs  rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/cgroup                                                                     cgroup2                                                     cgroup2     rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/fs/pstore                                                                     pstore                                                      pstore      rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/firmware/efi/efivars                                                          efivarfs                                                    efivarfs    rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/bpf                                                                        none                                                        bpf         rw,nosuid,nodev,noexec,relatime,mode=700
| |-/sys/fs/selinux                                                                    selinuxfs                                                   selinuxfs   rw,nosuid,noexec,relatime
| |-/sys/kernel/debug                                                                  debugfs                                                     debugfs     rw,nosuid,nodev,noexec,relatime,seclabel
| | `-/sys/kernel/debug/tracing                                                        tracefs                                                     tracefs     rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/kernel/tracing                                                                tracefs                                                     tracefs     rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/fs/fuse/connections                                                           fusectl                                                     fusectl     rw,nosuid,nodev,noexec,relatime
| `-/sys/kernel/config                                                                 configfs                                                    configfs    rw,nosuid,nodev,noexec,relatime
|-/tmp                                                                                 rpool/redhat/root[/tmp]                                     zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| `-/tmp                                                                               rpool/redhat/root[/tmp/snap.lxd/tmp]                        zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/var/snap                                                                            rpool/redhat/root/var/snap                                  zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/var/snap/lxd/common/lxd/storage-pools                                             rpool/redhat/root/var/snap[/lxd/common/lxd/storage-pools]   zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| | `-/var/snap/lxd/common/lxd/storage-pools/lxd_pool/containers/lxd-dashboard         user-pool/-USERDATA/littleSnap/lxd/containers/lxd-dashboard zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/var/snap/lxd/common/lxd/devices                                                   rpool/redhat/root/var/snap[/lxd/common/lxd/devices]         zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/var/snap/lxd/common/ns                                                            tmpfs                                                       tmpfs       rw,relatime,seclabel,size=1024k,mode=700,inode64
| |-/var/snap/lxd/common/shmounts                                                      tmpfs                                                       tmpfs       rw,relatime,seclabel,size=1024k,mode=711,inode64
| | |-/var/snap/lxd/common/shmounts/lxcfs                                              lxcfs                                                       fuse.lxcfs  rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
| | `-/var/snap/lxd/common/shmounts/instances                                          tmpfs                                                       tmpfs       rw,relatime,seclabel,size=100k,mode=711,inode64
| `-/var/snap/lxd/common/lxd/devlxd                                                    tmpfs                                                       tmpfs       rw,relatime,seclabel,size=100k,mode=755,inode64
|-/var/lib/snapd                                                                       rpool/redhat/root/var/lib/snapd                             zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/var/lib/snapd/snap/snapd/17339                                                    /dev/loop0                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/var/lib/snapd/snap/lxd/23687                                                      /dev/loop1                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/var/lib/snapd/snap/snapd/17032                                                    /dev/loop2                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/var/lib/snapd/snap/core20/1627                                                    /dev/loop3                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| `-/var/lib/snapd/hostfs                                                              rpool/redhat/root/var/lib/snapd[/hostfs]                    zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|   `-/var/lib/snapd/hostfs                                                            rpool/redhat/root                                           zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/run                                                      tmpfs                                                       tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
|     | |-/var/lib/snapd/hostfs/run/user/4261                                          tmpfs                                                       tmpfs       rw,nosuid,nodev,relatime,seclabel,size=384948k,nr_inodes=96237,mode=700,uid=4261,gid=100,inode64
|     | `-/var/lib/snapd/hostfs/run/snapd/ns                                           tmpfs[/snapd/ns]                                            tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
|     |-/var/lib/snapd/hostfs/boot/efis/scsi-33001237923792379-part2                   /dev/sda2                                                   vfat        rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro
|     |-/var/lib/snapd/hostfs/boot                                                     bpool/redhat/boot                                           zfs         rw,nodev,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/-USERDATA                                                user-pool/-USERDATA                                         zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |-/var/lib/snapd/hostfs/-USERDATA/noSnap                                       user-pool/-USERDATA/noSnap                                  zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | | `-/var/lib/snapd/hostfs/-USERDATA/noSnap/libvirt/swapStoragePool             user-pool/-USERDATA/noSnap/libvirt/swapStoragePool          zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |-/var/lib/snapd/hostfs/-USERDATA/littleSnap                                   user-pool/-USERDATA/littleSnap                              zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | | `-/var/lib/snapd/hostfs/-USERDATA/littleSnap/libvirt                         user-pool/-USERDATA/littleSnap/libvirt                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |   |-/var/lib/snapd/hostfs/-USERDATA/littleSnap/libvirt/iso-pool              user-pool/-USERDATA/littleSnap/libvirt/iso-pool             zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |   `-/var/lib/snapd/hostfs/-USERDATA/littleSnap/libvirt/storagePool           user-pool/-USERDATA/littleSnap/libvirt/storagePool          zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |     `-/var/lib/snapd/hostfs/-USERDATA/littleSnap/libvirt/storagePool/default user-pool/-USERDATA/littleSnap/libvirt/storagePool/default  zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | `-/var/lib/snapd/hostfs/-USERDATA/muchSnap                                     user-pool/-USERDATA/muchSnap                                zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |   |-/var/lib/snapd/hostfs/-USERDATA/muchSnap/docker                            user-pool/-USERDATA/muchSnap/docker                         zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |   `-/var/lib/snapd/hostfs/-USERDATA/muchSnap/resources                         user-pool/-USERDATA/muchSnap/resources                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/root                                                     rpool/redhat/root/root                                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/home                                                     rpool/redhat/root/home                                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/var/lib                                                  rpool/redhat/root/var/lib                                   zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |-/var/lib/snapd/hostfs/var/lib/NetworkManager                                 rpool/redhat/root/var/lib/NetworkManager                    zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |-/var/lib/snapd/hostfs/var/lib/nfs                                            rpool/redhat/NoSnap-var/lib/nfs                             zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | | `-/var/lib/snapd/hostfs/var/lib/nfs/rpc_pipefs                               sunrpc                                                      rpc_pipefs  rw,relatime
|     | |-/var/lib/snapd/hostfs/var/lib/docker                                         rpool/redhat/root/var/lib/docker                            zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | `-/var/lib/snapd/hostfs/var/lib/snapd                                          rpool/redhat/root/var/lib/snapd                             zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |   |-/var/lib/snapd/hostfs/var/lib/snapd/snap/snapd/17339                       /dev/loop0                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|     |   |-/var/lib/snapd/hostfs/var/lib/snapd/snap/lxd/23687                         /dev/loop1                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|     |   |-/var/lib/snapd/hostfs/var/lib/snapd/snap/snapd/17032                       /dev/loop2                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|     |   `-/var/lib/snapd/hostfs/var/lib/snapd/snap/core20/1627                       /dev/loop3                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|     |-/var/lib/snapd/hostfs/var/tmp                                                  rpool/redhat/NoSnap-var/tmp                                 zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/usr/local                                                rpool/redhat/root/usr/local                                 zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/var/cache                                                rpool/redhat/NoSnap-var/cache                               zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/var/spool                                                rpool/redhat/root/var/spool                                 zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/var/log                                                  rpool/redhat/root/var/log                                   zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/var/snap                                                 rpool/redhat/root/var/snap                                  zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | `-/var/lib/snapd/hostfs/var/snap/lxd/common/ns                                 tmpfs                                                       tmpfs       rw,relatime,seclabel,size=1024k,mode=700,inode64
|     |-/var/lib/snapd/hostfs/dev                                                      devtmpfs                                                    devtmpfs    rw,nosuid,seclabel,size=1906580k,nr_inodes=476645,mode=755,inode64
|     |-/var/lib/snapd/hostfs/proc                                                     proc                                                        proc        rw,nosuid,nodev,noexec,relatime
|     `-/var/lib/snapd/hostfs/sys                                                      sysfs                                                       sysfs       rw,nosuid,nodev,noexec,relatime,seclabel
|-/var/tmp                                                                             rpool/redhat/NoSnap-var/tmp                                 zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/run                                                                                 tmpfs                                                       tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
| |-/run/user/4261                                                                     tmpfs                                                       tmpfs       rw,nosuid,nodev,relatime,seclabel,size=384948k,nr_inodes=96237,mode=700,uid=4261,gid=100,inode64
| |-/run/snapd/ns                                                                      tmpfs[/snapd/ns]                                            tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
| |-/run/media                                                                         tmpfs[/media]                                               tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
| |-/run/netns                                                                         tmpfs[/netns]                                               tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
| `-/run                                                                               tmpfs                                                       tmpfs       rw,nosuid,nodev,relatime,seclabel,mode=755,inode64
|-/usr/lib/modules                                                                     rpool/redhat/root[/usr/lib/modules]                         zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/usr/lib/firmware                                                                    rpool/redhat/root[/usr/lib/firmware]                        zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/usr/src                                                                             rpool/redhat/root[/usr/src]                                 zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/var/log                                                                             rpool/redhat/root/var/log                                   zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/media                                                                               tmpfs[/media]                                               tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
|-/mnt                                                                                 rpool/redhat/root[/mnt]                                     zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/usr/lib/snapd                                                                       rpool/redhat/root[/usr/libexec/snapd]                       zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/snap                                                                                rpool/redhat/root/var/lib/snapd[/snap]                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/snap/snapd/17339                                                                  /dev/loop0                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/snap/lxd/23687                                                                    /dev/loop1                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/snap/snapd/17032                                                                  /dev/loop2                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| `-/snap/core20/1627                                                                  /dev/loop3                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|-/usr/bin/kmod                                                                        /dev/loop1[/wrappers/kmod]                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|-/boot                                                                                bpool/redhat/boot                                           zfs         rw,nodev,relatime,seclabel,xattr,posixacl,casesensitive
`-/usr/share/misc                                                                      tmpfs                                                       tmpfs       rw,relatime,seclabel,mode=755,inode64
╰─

cola16 · October 24, 2022, 1:07am

Same mount setting on lxd, reboot works.

cola16 · October 25, 2022, 7:42am

pls, Somebody help me.
TT

tomp · October 25, 2022, 10:06am

Can you manually remove the directory, start the vm and then stop it (using lxc stop) and then get the contents of /var/snap/lxd/common/lxd/logs/lxd.log please.

cola16 · October 25, 2022, 11:41am

Thank you for your reply.

here is the all of log.

╰─ sudo cat /var/snap/lxd/common/lxd/logs/lxd.log                            ─╯
time="2022-10-22T01:37:53+09:00" level=warning msg="AppArmor support has been disabled because of lack of kernel support"
time="2022-10-22T01:37:53+09:00" level=warning msg=" - AppArmor support has been disabled, Disabled because of lack of kernel support"
time="2022-10-22T01:37:53+09:00" level=warning msg=" - Couldn't find the CGroup network priority controller, network priority will be ignored"
time="2022-10-25T20:39:08+09:00" level=warning msg="Unable to use virtio-fs for device, using 9p as a fallback" device=USERDATA driver=disk err="Architecture unsupported" instance=ubuntu-vm project=default
time="2022-10-25T20:39:08+09:00" level=warning msg="Unable to use virtio-fs for config drive, using 9p as a fallback" err="Architecture unsupported" instance=ubuntu-vm instanceType=virtual-machine project=default
time="2022-10-25T20:40:01+09:00" level=error msg="Failed to stop device" device=USERDATA err="Failed unmounting \"/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA\": Failed to unmount '/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA': device or resource busy" instance=ubuntu-vm instanceType=virtual-machine project=default

tomp · October 25, 2022, 12:25pm

Please can you show lxc storage show <pool>

cola16 · October 26, 2022, 12:29am

Only the lxd-dashboard container is running.
Other containers or vm are stopped.
In addition, this pool(lxd_pool) is not mounted, so I assumed that there will be no impact to the 9p storage.

If you think, this pool is a reason, do I try recreating lxd_pool in another zfs pool?

╰─ lxc storage show lxd_pool
config:
  source: user-pool/-USERDATA/littleSnap/lxd
  volatile.initial_source: user-pool/-USERDATA/littleSnap/lxd
  zfs.pool_name: user-pool/-USERDATA/littleSnap/lxd
description: ""
name: lxd_pool
driver: zfs
used_by:
- /1.0/images/46dc39a9cab5844795e7a8c1cbcaf42b0095ab8e28c7e27a974c2aa56ec0d4ae
- /1.0/images/5582f7885f47195c6c813309022dfeedc1cb6867b4542cbbf21c556cd0e3339f
- /1.0/images/61fe259f5411ed27962fe5dc2a7bf96e7991766d4a447933c54d327e267cdbff
- /1.0/instances/OL
- /1.0/instances/OL/snapshots/before-webmin
- /1.0/instances/OL/snapshots/webmin-zsh
- /1.0/instances/OL/snapshots/zsh-docker_run
- /1.0/instances/lxd-dashboard
- /1.0/instances/ubuntu-vm
- /1.0/profiles/default
- /1.0/storage-pools/lxd_pool/volumes/custom/test
status: Created
locations:
- none
╰─ zfs get mounted user-pool/-USERDATA/littleSnap/lxd
NAME                                PROPERTY  VALUE    SOURCE
user-pool/-USERDATA/littleSnap/lxd  mounted   no       -

tomp · October 26, 2022, 5:20am

I suspect the problem with unmounts is because the disk USERDATA source is /-USERDATA which will also contain the mount for the instance’s root disk.

Also because the USERDATA is recursive, it will be recreating any additional sub-mounts as well.

Please do not share the storage pool datasets/directories into an instance.

cola16 · October 26, 2022, 8:26am

The directory is not mounted.
The zfs pool(not lxc pool, user-pool/-USERDATA/littleSnap/lxd) is under ‘user-pool/-USERDATA/’
but the pool does not on ‘/-USERDATA/littleSnap/lxd’

The “recursive” means that mount all of sub-“mounts”, not sub-volume
However, the zfs volume is not mounted, not mountable.
So I did not share the storage pool into instance.

Also lxd doesn’t mount the zfs subvolume, mount just sub-mounts.
There are no lxd volume

cola16:

╰─ lxc stop ubuntu-vm
─ sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- findmnt
TARGET                                                                                                                   SOURCE                                                      FSTYPE      OPTIONS
/                                                                                                                        /dev/loop3                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|-/dev                                                                                                                   devtmpfs                                                    devtmpfs    rw,nosuid,seclabel,size=1906580k,nr_inodes=476645,mode=755,inode64
| |-/dev/ptmx                                                                                                            devpts[/ptmx]                                               devpts      rw,relatime,seclabel,gid=5,mode=620,ptmxmode=666
| |-/dev/shm                                                                                                             tmpfs                                                       tmpfs       rw,nosuid,nodev,seclabel,inode64
| |-/dev/pts                                                                                                             devpts                                                      devpts      rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000
| | `-/dev/pts                                                                                                           devpts                                                      devpts      rw,relatime,seclabel,gid=5,mode=620,ptmxmode=666
| |-/dev/hugepages                                                                                                       hugetlbfs                                                   hugetlbfs   rw,relatime,seclabel,pagesize=2M
| `-/dev/mqueue                                                                                                          mqueue                                                      mqueue      rw,nosuid,nodev,noexec,relatime,seclabel
|-/etc                                                                                                                   rpool/redhat/root[/etc]                                     zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/etc/nsswitch.conf                                                                                                   /dev/loop3[/etc/nsswitch.conf]                              squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/etc/ssl                                                                                                             /dev/loop3[/etc/ssl]                                        squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| `-/etc                                                                                                                 tmpfs                                                       tmpfs       rw,relatime,seclabel,mode=755,inode64
|-/home                                                                                                                  rpool/redhat/root/home                                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/root                                                                                                                  rpool/redhat/root/root                                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/proc                                                                                                                  proc                                                        proc        rw,nosuid,nodev,noexec,relatime
| |-/proc/sys/fs/binfmt_misc                                                                                             systemd-1                                                   autofs      rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=12903
| | `-/proc/sys/fs/binfmt_misc                                                                                           binfmt_misc                                                 binfmt_misc rw,nosuid,nodev,noexec,relatime
| `-/proc/fs/nfsd                                                                                                        nfsd                                                        nfsd        rw,relatime
|-/sys                                                                                                                   sysfs                                                       sysfs       rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/kernel/security                                                                                                 securityfs                                                  securityfs  rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/cgroup                                                                                                       cgroup2                                                     cgroup2     rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/fs/pstore                                                                                                       pstore                                                      pstore      rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/firmware/efi/efivars                                                                                            efivarfs                                                    efivarfs    rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/bpf                                                                                                          none                                                        bpf         rw,nosuid,nodev,noexec,relatime,mode=700
| |-/sys/fs/selinux                                                                                                      selinuxfs                                                   selinuxfs   rw,nosuid,noexec,relatime
| |-/sys/kernel/debug                                                                                                    debugfs                                                     debugfs     rw,nosuid,nodev,noexec,relatime,seclabel
| | `-/sys/kernel/debug/tracing                                                                                          tracefs                                                     tracefs     rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/kernel/tracing                                                                                                  tracefs                                                     tracefs     rw,nosuid,nodev,noexec,relatime,seclabel
| |-/sys/fs/fuse/connections                                                                                             fusectl                                                     fusectl     rw,nosuid,nodev,noexec,relatime
| `-/sys/kernel/config                                                                                                   configfs                                                    configfs    rw,nosuid,nodev,noexec,relatime
|-/tmp                                                                                                                   rpool/redhat/root[/tmp]                                     zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| `-/tmp                                                                                                                 rpool/redhat/root[/tmp/snap.lxd/tmp]                        zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/var/snap                                                                                                              rpool/redhat/root/var/snap                                  zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/var/snap/lxd/common/lxd/storage-pools                                                                               rpool/redhat/root/var/snap[/lxd/common/lxd/storage-pools]   zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| | `-/var/snap/lxd/common/lxd/storage-pools/lxd_pool/containers/lxd-dashboard                                           user-pool/-USERDATA/littleSnap/lxd/containers/lxd-dashboard zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/var/snap/lxd/common/lxd/devices                                                                                     rpool/redhat/root/var/snap[/lxd/common/lxd/devices]         zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| | `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA                                                user-pool/-USERDATA                                         zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |   |-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/noSnap                                       user-pool/-USERDATA/noSnap                                  zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |   | `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/noSnap/libvirt/swapStoragePool             user-pool/-USERDATA/noSnap/libvirt/swapStoragePool          zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |   |-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/littleSnap                                   user-pool/-USERDATA/littleSnap                              zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |   | `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/littleSnap/libvirt                         user-pool/-USERDATA/littleSnap/libvirt                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |   |   |-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/littleSnap/libvirt/iso-pool              user-pool/-USERDATA/littleSnap/libvirt/iso-pool             zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |   |   `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/littleSnap/libvirt/storagePool           user-pool/-USERDATA/littleSnap/libvirt/storagePool          zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |   |     `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/littleSnap/libvirt/storagePool/default user-pool/-USERDATA/littleSnap/libvirt/storagePool/default  zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |   `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/muchSnap                                     user-pool/-USERDATA/muchSnap                                zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |     |-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/muchSnap/docker                            user-pool/-USERDATA/muchSnap/docker                         zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |     `-/var/snap/lxd/common/lxd/devices/ubuntu-vm/disk.USERDATA.--USERDATA/muchSnap/resources                         user-pool/-USERDATA/muchSnap/resources                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/var/snap/lxd/common/ns                                                                                              tmpfs                                                       tmpfs       rw,relatime,seclabel,size=1024k,mode=700,inode64
| |-/var/snap/lxd/common/shmounts                                                                                        tmpfs                                                       tmpfs       rw,relatime,seclabel,size=1024k,mode=711,inode64
| | |-/var/snap/lxd/common/shmounts/lxcfs                                                                                lxcfs                                                       fuse.lxcfs  rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
| | `-/var/snap/lxd/common/shmounts/instances                                                                            tmpfs                                                       tmpfs       rw,relatime,seclabel,size=100k,mode=711,inode64
| `-/var/snap/lxd/common/lxd/devlxd                                                                                      tmpfs                                                       tmpfs       rw,relatime,seclabel,size=100k,mode=755,inode64
|-/var/lib/snapd                                                                                                         rpool/redhat/root/var/lib/snapd                             zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/var/lib/snapd/snap/snapd/17339                                                                                      /dev/loop0                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/var/lib/snapd/snap/lxd/23687                                                                                        /dev/loop1                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/var/lib/snapd/snap/snapd/17032                                                                                      /dev/loop2                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/var/lib/snapd/snap/core20/1627                                                                                      /dev/loop3                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| `-/var/lib/snapd/hostfs                                                                                                rpool/redhat/root/var/lib/snapd[/hostfs]                    zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|   `-/var/lib/snapd/hostfs                                                                                              rpool/redhat/root                                           zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/run                                                                                        tmpfs                                                       tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
|     | |-/var/lib/snapd/hostfs/run/user/4261                                                                            tmpfs                                                       tmpfs       rw,nosuid,nodev,relatime,seclabel,size=384948k,nr_inodes=96237,mode=700,uid=4261,gid=100,inode64
|     | `-/var/lib/snapd/hostfs/run/snapd/ns                                                                             tmpfs[/snapd/ns]                                            tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
|     |-/var/lib/snapd/hostfs/boot/efis/scsi-33001237923792379-part2                                                     /dev/sda2                                                   vfat        rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro
|     |-/var/lib/snapd/hostfs/boot                                                                                       bpool/redhat/boot                                           zfs         rw,nodev,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/-USERDATA                                                                                  user-pool/-USERDATA                                         zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |-/var/lib/snapd/hostfs/-USERDATA/noSnap                                                                         user-pool/-USERDATA/noSnap                                  zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | | `-/var/lib/snapd/hostfs/-USERDATA/noSnap/libvirt/swapStoragePool                                               user-pool/-USERDATA/noSnap/libvirt/swapStoragePool          zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |-/var/lib/snapd/hostfs/-USERDATA/littleSnap                                                                     user-pool/-USERDATA/littleSnap                              zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | | `-/var/lib/snapd/hostfs/-USERDATA/littleSnap/libvirt                                                           user-pool/-USERDATA/littleSnap/libvirt                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |   |-/var/lib/snapd/hostfs/-USERDATA/littleSnap/libvirt/iso-pool                                                user-pool/-USERDATA/littleSnap/libvirt/iso-pool             zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |   `-/var/lib/snapd/hostfs/-USERDATA/littleSnap/libvirt/storagePool                                             user-pool/-USERDATA/littleSnap/libvirt/storagePool          zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |     `-/var/lib/snapd/hostfs/-USERDATA/littleSnap/libvirt/storagePool/default                                   user-pool/-USERDATA/littleSnap/libvirt/storagePool/default  zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | `-/var/lib/snapd/hostfs/-USERDATA/muchSnap                                                                       user-pool/-USERDATA/muchSnap                                zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |   |-/var/lib/snapd/hostfs/-USERDATA/muchSnap/docker                                                              user-pool/-USERDATA/muchSnap/docker                         zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |   `-/var/lib/snapd/hostfs/-USERDATA/muchSnap/resources                                                           user-pool/-USERDATA/muchSnap/resources                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/root                                                                                       rpool/redhat/root/root                                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/home                                                                                       rpool/redhat/root/home                                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/var/lib                                                                                    rpool/redhat/root/var/lib                                   zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |-/var/lib/snapd/hostfs/var/lib/NetworkManager                                                                   rpool/redhat/root/var/lib/NetworkManager                    zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | |-/var/lib/snapd/hostfs/var/lib/nfs                                                                              rpool/redhat/NoSnap-var/lib/nfs                             zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | | `-/var/lib/snapd/hostfs/var/lib/nfs/rpc_pipefs                                                                 sunrpc                                                      rpc_pipefs  rw,relatime
|     | |-/var/lib/snapd/hostfs/var/lib/docker                                                                           rpool/redhat/root/var/lib/docker                            zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | `-/var/lib/snapd/hostfs/var/lib/snapd                                                                            rpool/redhat/root/var/lib/snapd                             zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |   |-/var/lib/snapd/hostfs/var/lib/snapd/snap/snapd/17339                                                         /dev/loop0                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|     |   |-/var/lib/snapd/hostfs/var/lib/snapd/snap/lxd/23687                                                           /dev/loop1                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|     |   |-/var/lib/snapd/hostfs/var/lib/snapd/snap/snapd/17032                                                         /dev/loop2                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|     |   `-/var/lib/snapd/hostfs/var/lib/snapd/snap/core20/1627                                                         /dev/loop3                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|     |-/var/lib/snapd/hostfs/var/tmp                                                                                    rpool/redhat/NoSnap-var/tmp                                 zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/usr/local                                                                                  rpool/redhat/root/usr/local                                 zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/var/cache                                                                                  rpool/redhat/NoSnap-var/cache                               zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/var/spool                                                                                  rpool/redhat/root/var/spool                                 zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/var/log                                                                                    rpool/redhat/root/var/log                                   zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     |-/var/lib/snapd/hostfs/var/snap                                                                                   rpool/redhat/root/var/snap                                  zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|     | `-/var/lib/snapd/hostfs/var/snap/lxd/common/ns                                                                   tmpfs                                                       tmpfs       rw,relatime,seclabel,size=1024k,mode=700,inode64
|     |-/var/lib/snapd/hostfs/dev                                                                                        devtmpfs                                                    devtmpfs    rw,nosuid,seclabel,size=1906580k,nr_inodes=476645,mode=755,inode64
|     |-/var/lib/snapd/hostfs/proc                                                                                       proc                                                        proc        rw,nosuid,nodev,noexec,relatime
|     `-/var/lib/snapd/hostfs/sys                                                                                        sysfs                                                       sysfs       rw,nosuid,nodev,noexec,relatime,seclabel
|-/var/tmp                                                                                                               rpool/redhat/NoSnap-var/tmp                                 zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/run                                                                                                                   tmpfs                                                       tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
| |-/run/user/4261                                                                                                       tmpfs                                                       tmpfs       rw,nosuid,nodev,relatime,seclabel,size=384948k,nr_inodes=96237,mode=700,uid=4261,gid=100,inode64
| |-/run/snapd/ns                                                                                                        tmpfs[/snapd/ns]                                            tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
| |-/run/media                                                                                                           tmpfs[/media]                                               tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
| |-/run/netns                                                                                                           tmpfs[/netns]                                               tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
| `-/run                                                                                                                 tmpfs                                                       tmpfs       rw,nosuid,nodev,relatime,seclabel,mode=755,inode64
|-/usr/lib/modules                                                                                                       rpool/redhat/root[/usr/lib/modules]                         zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/usr/lib/firmware                                                                                                      rpool/redhat/root[/usr/lib/firmware]                        zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/usr/src                                                                                                               rpool/redhat/root[/usr/src]                                 zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/var/log                                                                                                               rpool/redhat/root/var/log                                   zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/media                                                                                                                 tmpfs[/media]                                               tmpfs       rw,nosuid,nodev,seclabel,size=769904k,nr_inodes=819200,mode=755,inode64
|-/mnt                                                                                                                   rpool/redhat/root[/mnt]                                     zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/usr/lib/snapd                                                                                                         rpool/redhat/root[/usr/libexec/snapd]                       zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
|-/snap                                                                                                                  rpool/redhat/root/var/lib/snapd[/snap]                      zfs         rw,relatime,seclabel,xattr,posixacl,casesensitive
| |-/snap/snapd/17339                                                                                                    /dev/loop0                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/snap/lxd/23687                                                                                                      /dev/loop1                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| |-/snap/snapd/17032                                                                                                    /dev/loop2                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
| `-/snap/core20/1627                                                                                                    /dev/loop3                                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|-/usr/bin/kmod                                                                                                          /dev/loop1[/wrappers/kmod]                                  squashfs    ro,nodev,relatime,context=system_u:object_r:snappy_snap_t:s0,errors=continue
|-/boot                                                                                                                  bpool/redhat/boot                                           zfs         rw,nodev,relatime,seclabel,xattr,posixacl,casesensitive
`-/usr/share/misc

╰─ mount | grep USERD
user-pool/-USERDATA on /-USERDATA type zfs (rw,relatime,seclabel,xattr,posixacl,casesensitive)
user-pool/-USERDATA/muchSnap on /-USERDATA/muchSnap type zfs (rw,relatime,seclabel,xattr,posixacl,casesensitive)
user-pool/-USERDATA/littleSnap on /-USERDATA/littleSnap type zfs (rw,relatime,seclabel,xattr,posixacl,casesensitive)
user-pool/-USERDATA/noSnap on /-USERDATA/noSnap type zfs (rw,relatime,seclabel,xattr,posixacl,casesensitive)
user-pool/-USERDATA/muchSnap/resources on /-USERDATA/muchSnap/resources type zfs (rw,relatime,seclabel,xattr,posixacl,casesensitive)
user-pool/-USERDATA/muchSnap/docker on /-USERDATA/muchSnap/docker type zfs (rw,relatime,seclabel,xattr,posixacl,casesensitive)
user-pool/-USERDATA/littleSnap/libvirt on /-USERDATA/littleSnap/libvirt type zfs (rw,relatime,seclabel,xattr,posixacl,casesensitive)
user-pool/-USERDATA/noSnap/libvirt/swapStoragePool on /-USERDATA/noSnap/libvirt/swapStoragePool type zfs (rw,relatime,seclabel,xattr,posixacl,casesensitive)
user-pool/-USERDATA/littleSnap/libvirt/storagePool on /-USERDATA/littleSnap/libvirt/storagePool type zfs (rw,relatime,seclabel,xattr,posixacl,casesensitive)
user-pool/-USERDATA/littleSnap/libvirt/iso-pool on /-USERDATA/littleSnap/libvirt/iso-pool type zfs (rw,relatime,seclabel,xattr,posixacl,casesensitive)
user-pool/-USERDATA/littleSnap/libvirt/storagePool/default on /-USERDATA/littleSnap/libvirt/storagePool/default type zfs (rw,relatime,seclabel,xattr,posixacl,casesensitive)
user-pool/-USERDATA/littleSnap/lxd on /-lxdpool type zfs (rw,noatime,seclabel,xattr,posixacl,casesensitive)

cola16 · October 26, 2022, 8:44am

Fortunately, that wasn’t the problem.
This time, mounting the user-pool/-USERDATA is same as the previous,
but the lxc storage pool is user-pool/-lxd, not user-pool/-USERDATA/littlesnap/lxd.

Still I have same problem.

cola16 · November 2, 2022, 4:48am

I thought of LXD’s Virtualmachine as an alternative to libVirt that can’t use snapshots in ARM architecture.
Unfortunately, I can’t use LXD vm.