Failed to create mount directory (Snap, BTRFS)

LXD
1

(Snap, BTRFS, Arch host, Ubuntu container)
Everything was working great till I restored a two hour old Timeshift snapshot. Now when I try to start a container, I get:

Failed preparing container for start: Failed to create mount directory "/var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/containers/unifi-controller": mkdir /var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/containers/unifi-controller: no such file or directoryd text.

$ lxc list
(Shows all container, none missing)

$ lxc storage list
LXD-BTRFS btrfs /var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS 6

$ sudo btrfs su list /
ID 257 gen 586918 top level 5 path @home
ID 258 gen 244981 top level 5 path @swap
ID 3341 gen 586216 top level 5 path timeshift-btrfs/snapshots/2021-03-23_13-19-56/@
ID 3342 gen 586485 top level 3341 path timeshift-btrfs/snapshots/2021-03-23_13-19-56/@/var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS
ID 3345 gen 583149 top level 3342 path timeshift-btrfs/snapshots/2021-03-23_13-19-56/@/var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/containers/next-server
ID 3348 gen 578660 top level 3342 path timeshift-btrfs/snapshots/2021-03-23_13-19-56/@/var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/containers/minecraft
ID 3411 gen 585356 top level 3342 path timeshift-btrfs/snapshots/2021-03-23_13-19-56/@/var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/containers/unifi-controller
ID 3613 gen 578653 top level 3342 path timeshift-btrfs/snapshots/2021-03-23_13-19-56/@/var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/containers-snapshots/minecraft/snap1
ID 3615 gen 537039 top level 3342 path timeshift-btrfs/snapshots/2021-03-23_13-19-56/@/var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/containers-snapshots/unifi-controller/
snap1
ID 3619 gen 583149 top level 3342 path timeshift-btrfs/snapshots/2021-03-23_13-19-56/@/var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/containers/next-main
ID 3655 gen 537364 top level 3342 path timeshift-btrfs/snapshots/2021-03-23_13-19-56/@/var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/containers-snapshots/next-server/snap1
ID 3729 gen 585109 top level 5 path timeshift-btrfs/snapshots/2021-03-21_15-00-02/@
ID 3759 gen 585110 top level 5 path timeshift-btrfs/snapshots/2021-03-22_15-00-01/@
ID 3765 gen 578238 top level 3342 path timeshift-btrfs/snapshots/2021-03-23_13-19-56/@/var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/images/077af5d40a93b38ea51709a0b97dad7
72cbe5ace8cc98636c6cee2659fbf96a4
ID 3786 gen 586918 top level 5 path @
ID 3787 gen 586216 top level 5 path timeshift-btrfs/snapshots/2021-03-23_15-00-01/@
ID 3788 gen 586216 top level 5 path timeshift-btrfs/snapshots/2021-03-23_16-00-01/@
ID 3789 gen 586216 top level 5 path timeshift-btrfs/snapshots/2021-03-23_17-00-01/@
ID 3790 gen 586216 top level 5 path timeshift-btrfs/snapshots/2021-03-23_18-00-02/@
ID 3791 gen 586216 top level 5 path timeshift-btrfs/snapshots/2021-03-23_19-00-02/@
ID 3792 gen 586216 top level 5 path timeshift-btrfs/snapshots/2021-03-23_20-00-01/@
ID 3793 gen 586598 top level 5 path timeshift-btrfs/snapshots/2021-03-23_21-00-01/@

$ cat /proc/self/mountinfo | grep lxd
350 31 7:1 / /var/lib/snapd/snap/lxd/19823 ro,nodev,relatime shared:192 - squashfs /dev/loop1 ro
357 31 7:2 / /var/lib/snapd/snap/lxd/19766 ro,nodev,relatime shared:196 - squashfs /dev/loop2 ro
852 832 0:4 mnt:[4026532856] /run/snapd/ns/lxd.mnt rw - nsfs nsfs rw
1278 31 0:86 / /var/snap/lxd/common/ns rw,relatime - tmpfs tmpfs rw,size=1024k,mode=700,inode64
1301 1278 0:4 mnt:[4026532833] /var/snap/lxd/common/ns/shmounts rw - nsfs nsfs rw
1256 1278 0:4 mnt:[4026532856] /var/snap/lxd/common/ns/mntns rw - nsfs nsfs rw
1190 31 0:27 /timeshift-btrfs/snapshots/2021-03-23_13-19-56/@/var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS /var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS rw,relatime sh
ared:573 - btrfs /dev/nvme0n1p2 rw,compress=zstd:3,ssd,space_cache,user_subvol_rm_allowed,autodefrag,commit=120,subvolid=3342,subvol=/timeshift-btrfs/snapshots/2021-03-23_13
-19-56/@/var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS
1449 1190 0:27 /timeshift-btrfs/snapshots/2021-03-23_13-19-56/@/var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/images/077af5d40a93b38ea51709a0b97dad772cbe5ace8cc98636c6cee2
659fbf96a4 /var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/images/077af5d40a93b38ea51709a0b97dad772cbe5ace8cc98636c6cee2659fbf96a4 rw,relatime shared:602 - btrfs /dev/nvme0
n1p2 rw,compress=zstd:3,ssd,space_cache,user_subvol_rm_allowed,autodefrag,commit=120,subvolid=3765,subvol=/timeshift-btrfs/snapshots/2021-03-23_13-19-56/@/var/snap/lxd/commo
n/lxd/storage-pools/LXD-BTRFS/images/077af5d40a93b38ea51709a0b97dad772cbe5ace8cc98636c6cee2659fbf96a4

When I manually mount my various snapshots to a temporary mount point I can still see the data is there, but am not sure how to recover and restore my working system. Any help you can provide would be greatly appreciated! Thank you in advance for your help!

Cheers,
Pete

2

Try nsenter --mount=/run/snapd/ns/lxd.mnt and then see if things look good from that point of view as that’s what LXD actually sees.

If you need the btrfs tools from this environment, you can get them through /snap/lxd/current/bin/btrfs

3

I am sincerely sorry to ask for it, but I really need more basic/explicit guidance. Running nsenter --mount=/run/snapd/ns/lxd.mnt puts me into -bash-4.4#, then what? Not sure what to do then?

4

Then go look at /var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS for what’s there.

5

# /var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS

Shows nothing there. This is from my normal prompt, not nsenter.

6

That’s normal

7

So FYI, I rebooted to start with a “clean” state before the last command. So now,
# cat /proc/self/mountinfo | grep lxd
350 31 7:2 / /var/lib/snapd/snap/lxd/19823 ro,nodev,relatime shared:192 - squashfs /dev/loop2 ro
364 31 7:3 / /var/lib/snapd/snap/lxd/19766 ro,nodev,relatime shared:200 - squashfs /dev/loop3 ro
852 832 0:4 mnt:[4026532856] /run/snapd/ns/lxd.mnt rw - nsfs nsfs rw
1278 31 0:88 / /var/snap/lxd/common/ns rw,relatime - tmpfs tmpfs rw,size=1024k,mode=700,inode64
1301 1278 0:4 mnt:[4026532833] /var/snap/lxd/common/ns/shmounts rw - nsfs nsfs rw
1256 1278 0:4 mnt:[4026532856] /var/snap/lxd/common/ns/mntns rw - nsfs nsfs rw

All else is same as originally reported.

8

Run lxc info to make sure LXD is running, the use the nsenter command I gave you to go see what’s mounted in the LXD mount namespace.

/proc/self/mountinfo from the host’s point of view will always be quite boring and should never show your storage pool, only the information inside the mount namespace matters.

9

$ lxc info
Long list of attributes given which I assume to mean LXD is running.

Ran # nsenter --mount=/run/snapd/ns/lxd.mnt am in its shell. What command do I now run to check mounts from its perspective?

10
  • ls -lh /var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/
  • ls -lh /var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/containers/

Would be a good place to start.

11

ls command not found.

12

Try /bin/ls maybe? Not sure why your system wouldn’t have ls though…

13 
-bash-4.4# bin/ls -lh /var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/   
total 0
-bash-4.4# bin/ls -lh /var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/containers/
bin/ls: cannot access '/var/snap/lxd/common/lxd/storage-pools/LXD-BTRFS/containers/': No such file or directory

I have ls aliased to exa in my .bashrc, that may have been the issue.

14

Can you actually run what I’m telling you to run please?
I wrote /bin/ls, not bin/ls.

15

Sorry! Re-ran it correctly this time, same result.

16

Okay, so that’s suggesting it’s not mounted somehow.
Can you show cat /proc/$(cat /var/snap/lxd/common/lxd.pid)/mountinfo

17

cat command not found. cat is another one that I have aliased, its aliased to bat. What would be the full path to it?

18

Probably /bin/cat

19

Tried /bin/cat and /usr/bin/cat ? Not found.

20 
stgraber@castiana:~$ sudo -i
root@castiana:~# nsenter --mount=/run/snapd/ns/lxd.mnt
root@castiana:/# /bin/cat /proc/$(/bin/cat /var/snap/lxd/common/lxd.pid)/mountinfo

Works perfectly fine here.