Hello,
lxd is running through snap on Ubuntu 20.04 with zfs.
Channel: 4.0/stable: 4.0.9-8e2046b
I cannot make a backup of a specific container:
$ lxc export st /mnt/wbub/st-$(date +%Y-%m-%d).tar.gz
Error: Create backup: Backup create: Snapshot "snap23" in storage but not expected
Following Can't delete container (on btrfs): "Cannot remove a volume that has snapshots", there is only one relevant line in sql storage_volumes:
$ lxd sql global "SELECT * FROM storage_volumes"
...
| 89 | st | 1 | 1 | 0 | | 1 |
...
and no corresponding directory in the snapshots folder:
$ sudo ls -lah /var/snap/lxd/common/lxd/storage-pools/lxd/containers-snapshots/st
total 64K
drwx------ 16 root root 4.0K Jun 7 06:55 .
drwx--x--x 6 root root 4.0K Aug 10 2021 ..
d--x------ 2 root root 4.0K May 25 03:14 snap340
d--x------ 2 root root 4.0K May 26 03:14 snap341
d--x------ 2 root root 4.0K May 27 03:14 snap342
d--x------ 2 root root 4.0K May 28 03:14 snap343
d--x------ 2 root root 4.0K May 29 03:14 snap344
d--x------ 2 root root 4.0K May 30 03:14 snap345
d--x------ 2 root root 4.0K May 31 03:14 snap346
d--x------ 2 root root 4.0K Jun 1 03:14 snap347
d--x------ 2 root root 4.0K Jun 2 03:14 snap348
d--x------ 2 root root 4.0K Jun 3 03:14 snap349
d--x------ 2 root root 4.0K Jun 4 03:14 snap350
d--x------ 2 root root 4.0K Jun 5 03:14 snap351
d--x------ 2 root root 4.0K Jun 6 03:14 snap352
d--x------ 2 root root 4.0K Jun 7 03:14 snap353
So, where can I find this snap23 to delete it?
Thanks in advance.
*added directory for ls
Oops, sorry:
$ sudo ls -lah /var/snap/lxd/common/lxd/storage-pools/lxd/containers-snapshots/st
total 64K
drwx------ 16 root root 4.0K Jun 7 06:55 .
drwx--x--x 6 root root 4.0K Aug 10 2021 ..
d--x------ 2 root root 4.0K May 25 03:14 snap340
d--x------ 2 root root 4.0K May 26 03:14 snap341
d--x------ 2 root root 4.0K May 27 03:14 snap342
d--x------ 2 root root 4.0K May 28 03:14 snap343
d--x------ 2 root root 4.0K May 29 03:14 snap344
d--x------ 2 root root 4.0K May 30 03:14 snap345
d--x------ 2 root root 4.0K May 31 03:14 snap346
d--x------ 2 root root 4.0K Jun 1 03:14 snap347
d--x------ 2 root root 4.0K Jun 2 03:14 snap348
d--x------ 2 root root 4.0K Jun 3 03:14 snap349
d--x------ 2 root root 4.0K Jun 4 03:14 snap350
d--x------ 2 root root 4.0K Jun 5 03:14 snap351
d--x------ 2 root root 4.0K Jun 6 03:14 snap352
d--x------ 2 root root 4.0K Jun 7 03:14 snap353
tomp
(Thomas Parrott)
4
What do you see when running:
sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- ls -lah /var/snap/lxd/common/lxd/storage-pools/lxd/containers-snapshots
warrick@wbub:~$ sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- ls -lah /var/snap/lxd/common/lxd/storage-pools/lxd/containers-snapshots
total 24K
drwx--x--x 6 root root 4.0K Aug 10 2021 .
drwx--x--x 10 root root 4.0K Jun 7 07:18 ..
drwx------ 16 root root 4.0K Jun 8 16:51 box
drwx------ 16 root root 4.0K Jun 8 16:15 ost
drwx------ 16 root root 4.0K Jun 9 03:14 st
drwx------ 9 root root 4.0K Jun 8 16:39 wg
warrick@wbub:~$ sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- ls -lah /var/snap/lxd/common/lxd/storage-pools/lxd/containers-snapshots/st/
total 64K
drwx------ 16 root root 4.0K Jun 9 03:14 .
drwx--x--x 6 root root 4.0K Aug 10 2021 ..
d--x------ 2 root root 4.0K May 27 03:14 snap342
d--x------ 2 root root 4.0K May 28 03:14 snap343
d--x------ 2 root root 4.0K May 29 03:14 snap344
d--x------ 2 root root 4.0K May 30 03:14 snap345
d--x------ 2 root root 4.0K May 31 03:14 snap346
d--x------ 2 root root 4.0K Jun 1 03:14 snap347
d--x------ 2 root root 4.0K Jun 2 03:14 snap348
d--x------ 2 root root 4.0K Jun 3 03:14 snap349
d--x------ 2 root root 4.0K Jun 4 03:14 snap350
d--x------ 2 root root 4.0K Jun 5 03:14 snap351
d--x------ 2 root root 4.0K Jun 6 03:14 snap352
d--x------ 2 root root 4.0K Jun 7 03:14 snap353
d--x------ 2 root root 4.0K Jun 8 03:14 snap354
d--x------ 2 root root 4.0K Jun 9 03:14 snap355
tomp
(Thomas Parrott)
6
Weird, looks like there may be a rogue mount lingering with the old directory in it.
Have you rebooted recently?
This has been an ongoing issue since 9-2021 and has persisted across reboots. I just never bothered to do anything about it until now.
tomp
(Thomas Parrott)
8
OK so the ZFS driver doesn’t use the directories to indicate which snapshots are available, it uses the zfs command.
Can you run this command please?
sudo zfs list -t snapshot
And there it is!
warrick@wbub:~$ sudo zfs list -t snapshot | grep snap23
lxd/containers/st@snapshot-snap23 1.28G - 1.60G -
After running zfs destroy to get rid of it, the export completes successfully.
Thank you Thomas! I did not think to check zfs.
1 Like
