LXD: Cannot find and remove snapshot to remove storage pool?

teward · April 7, 2020, 3:42am

I’ve got several ZFS storage pools. Unfortunately, the ZFS storage pool is… giving some issues when I am trying to remove ‘default’ storage pool since I have named storage pools specifically for containers or for VMs separately.

This is the output of lxc storage show default:

$ lxc storage show default
config:
  size: 100GB
  source: /var/snap/lxd/common/lxd/disks/default.img
  zfs.pool_name: default
description: ""
name: default
driver: zfs
used_by:
- /1.0/containers/local-dns/snapshots/snap0
status: Created
locations:
- none

Now I can’t locate that snapshot. It doesn’t exist. How can I ‘remove’ that snapshot/disk and then purge the ‘default’ store?

tomp · April 7, 2020, 12:17pm

Deleting the VM instance or its snapshot using lxc delete <instance> or lxc delete <instance>/<snapshot> will also remove the storage snapshot.

You can see them using lxc storage volume list <pool>

teward · April 7, 2020, 12:54pm

That’s the thing - the thing says that the snapshot doesnt exist when I try to delete it. I also cant delete the instance at the moment because its powering the DNS on my system.

In effect LXD says the snapshot doesnt exist yet it also says the snapshot exists at the same time.

tomp · April 7, 2020, 12:55pm

Can you show the commands you are running and the output please (helps to build up a clear picture - the more info the better).

Also please show the output of sudo zfs list -t snapshot please?

tomp · April 7, 2020, 12:56pm

And the output of lxc storage volume list <pool> please.

teward · April 7, 2020, 1:40pm

All requested commands, plus the attempt to delete the nonexistent snapshot. It’s almost like LXD and ZFS are out of sync and in an inconsistent state…

$ sudo zfs list -t snapshot
no datasets available

$ lxc storage volume list default
+----------------------+-----------------+-------------+---------+
|         TYPE         |      NAME       | DESCRIPTION | USED BY |
+----------------------+-----------------+-------------+---------+
| container (snapshot) | local-dns/snap0 |             | 1       |
+----------------------+-----------------+-------------+---------+

$ lxc storage show default
config:
  size: 100GB
  source: /var/snap/lxd/common/lxd/disks/default.img
  zfs.pool_name: default
description: ""
name: default
driver: zfs
used_by:
- /1.0/containers/local-dns/snapshots/snap0
status: Created
locations:
- none

$ lxc delete local-dns/snap0
Error: Failed to fetch snapshot "snap0" of instance "local-dns" in project "default": No such object

tomp · April 7, 2020, 1:42pm

Thanks, that is likely to be the issue.

Can you also show output of zfs list please?

Did you manually remove any ZFS volumes using the zfs command or get any error from LXD before this one?

teward · April 7, 2020, 1:44pm

Yeah earlier today actually the ZFS pool for default seemed to up and go away - the .img disappeared entirely so it was nonexistent. I’m guessing this is why it’s in an inconsistent state. Unfortunately, that poses some headaches. Any way to force LXD to forget the snapshot and then just remove the ‘default’ store from LXD?

teward · April 7, 2020, 1:50pm

$ zfs list
NAME                                                                     USED  AVAIL  REFER  MOUNTPOINT
default                                                                   84K  39.9M    24K  /default
lxd-containers                                                          13.6G   212G    24K  none
lxd-containers/containers                                               12.7G   212G    24K  none
lxd-containers/containers/dns-sinkhole                                   467M   212G   467M  /var/snap/lxd/common/lxd/storage-pools/lxd-containers/containers/dns-sinkhole
lxd-containers/containers/redacted0                                 507M   212G   507M  /var/snap/lxd/common/lxd/storage-pools/lxd-containers/containers/redacted0
lxd-containers/containers/docker                                        6.93G   212G  6.93G  /var/snap/lxd/common/lxd/storage-pools/lxd-containers/containers/docker
lxd-containers/containers/focal-proposed-testing                        1.10G   212G  1.10G  /var/snap/lxd/common/lxd/storage-pools/lxd-containers/containers/focal-proposed-testing
lxd-containers/containers/redacted1                                      525M   212G   525M  /var/snap/lxd/common/lxd/storage-pools/lxd-containers/containers/redacted1
lxd-containers/containers/local-db                                       573M   212G   573M  /var/snap/lxd/common/lxd/storage-pools/lxd-containers/containers/local-db
lxd-containers/containers/local-dns-backup                              7.29M   212G   948M  /var/snap/lxd/common/lxd/storage-pools/lxd-containers/containers/local-dns-backup
lxd-containers/containers/local-repo                                     481M   212G   481M  /var/snap/lxd/common/lxd/storage-pools/lxd-containers/containers/local-repo
lxd-containers/containers/lxd-dhcp-server                                583M   212G   583M  /var/snap/lxd/common/lxd/storage-pools/lxd-containers/containers/lxd-dhcp-server
lxd-containers/containers/powerdns                                      1.09G   212G  1.09G  /var/snap/lxd/common/lxd/storage-pools/lxd-containers/containers/powerdns
lxd-containers/containers/rabbitmq                                       528M   212G   528M  /var/snap/lxd/common/lxd/storage-pools/lxd-containers/containers/rabbitmq
lxd-containers/custom                                                     24K   212G    24K  none
lxd-containers/deleted                                                   951M   212G    24K  none
lxd-containers/deleted/containers                                        951M   212G    24K  none
lxd-containers/deleted/containers/a4439de7-09c4-4754-b63a-d3e90d1d6ffe   951M   212G   948M  /var/snap/lxd/common/lxd/storage-pools/lxd-containers/containers/local-dns
lxd-containers/deleted/custom                                             24K   212G    24K  none
lxd-containers/deleted/images                                             24K   212G    24K  none
lxd-containers/deleted/virtual-machines                                   24K   212G    24K  none
lxd-containers/images                                                     24K   212G    24K  none
lxd-containers/virtual-machines                                           24K   212G    24K  none
lxd-qemu                                                                 480K   187G    24K  none
lxd-qemu/containers                                                       24K   187G    24K  none
lxd-qemu/custom                                                           24K   187G    24K  none
lxd-qemu/deleted                                                         120K   187G    24K  none
lxd-qemu/deleted/containers                                               24K   187G    24K  none
lxd-qemu/deleted/custom                                                   24K   187G    24K  none
lxd-qemu/deleted/images                                                   24K   187G    24K  none
lxd-qemu/deleted/virtual-machines                                         24K   187G    24K  none
lxd-qemu/images                                                           24K   187G    24K  none
lxd-qemu/virtual-machines                                                 24K   187G    24K  none

Note that the redacted names are still present on the system I’m just hiding their names due to proprietary information and agreements.

tomp · April 7, 2020, 1:53pm

According to that the main volume for local-dns is also missing, so surprised that your container for local-dns is still running (you mentioned you couldn’t delete the container as it was in use)?

tomp · April 7, 2020, 1:58pm

You could try running:

lxd sql global 'delete from storage_volumes_snapshots where name="local-dns/snap0"'
lxc delete local-dns/snap0

teward · April 7, 2020, 2:00pm

i just nuked the container and the snapshot still exists - that’s why localdns doesn’t exist. local-dns-backup is running in its place.

teward · April 7, 2020, 2:02pm

$ lxd sql global 'delete from storage_volumes_snapshots where name="local-dns/snap0"'
Rows affected: 0

$ lxc delete local-dns/snap0
Error: not found

Looks a little like it’s still broke.

$ lxc storage volume list default
+----------------------+-----------------+-------------+---------+
|         TYPE         |      NAME       | DESCRIPTION | USED BY |
+----------------------+-----------------+-------------+---------+
| container (snapshot) | local-dns/snap0 |             | 1       |
+----------------------+-----------------+-------------+---------+

tomp · April 7, 2020, 2:03pm

Can you show the output of:

lxc sql global 'select * from storage_volumes_snapshots'

teward · April 7, 2020, 2:08pm

Invalid command, I assume you meant lxd so I used that:

$ lxd sql global 'select * from storage_volumes_snapshots'
+----+-------------------+------+-------------+-------------+
| id | storage_volume_id | name | description | expiry_date |
+----+-------------------+------+-------------+-------------+
+----+-------------------+------+-------------+-------------+

Something is realllllly inconsistent here.

tomp · April 7, 2020, 2:09pm

Thanks.

What version of LXD are you running?

teward · April 7, 2020, 2:09pm

Latest, 4.0, from Snaps.

$ lxc version
Client version: 4.0.0
Server version: 4.0.0

tomp · April 7, 2020, 2:12pm

@stgraber @freeekanayaka do you have any thoughts on this one, something appears strange with the database, there are snapshot results coming out of lxc storage volume list that dont show in lxd sql global 'select * from storage_volumes_snapshots'?

How does one get the r/o copy of the SQLITE DB for analysis with Snap?

stgraber · April 7, 2020, 2:33pm

lxd sql global .dump

Would be worth checking if maybe there’s an invalid record in storage_volumes.

tomp · April 7, 2020, 2:35pm

Thanks.

@teward can you send me the output of that command please, if you want to send it as a private message that is fine.