Delete snapshots on btrfs

LXD
1

Hello,

I am facing an issue where I am unable to delete the container/snapshot. My volume is based on block device on btrfs. This is what i think happened:

1.) lxc copy from remote to local
2.) Interrupt the copy making the copy on local stale
3.) Issued lxc delete -f
4.) However, snapshot remains on system /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/storpool1/containers-snapshots/testsnapshot
5.) However I am unable to delete these snapshots as it shows read-only. I have tried few suggestions from this forum without any success
6.) I tried to copy once again but since my source have snapshot on it, I manage to copy it specifying exact snapshot
7.) As expected, it is copied but lxc info does not show the snapshots i interrupted during my first copy
8.) lxc delete -f complains “Error: Error deleting storage volume: Cannot remove a volume that has snapshots”
9.) Tried the suggestion from @stgraber on Lxc export: "Snapshot ... in storage but not expected" and confirmed there is record.

Now i am stuck on how to clean it up. Does anyone have any idea? Thanks.

2

Can you show:

  • lxc version
  • lxc storage list
  • uname -a
3

Hi @stgraber ,

Blockquote
lxc@lxc4:~$ lxd version
4.20
lxc@lxc4:~$ lxc storage list
±----------±-------±-------------------------------------±------------±--------+
| NAME | DRIVER | SOURCE | DESCRIPTION | USED BY |
±----------±-------±-------------------------------------±------------±--------+
| storpool1 | btrfs | b199c777-7922-4338-8117-76ceed407c0e | | 56 |
±----------±-------±-------------------------------------±------------±--------+
lxc@lxc4:~$ uname -a
Linux lxc4 5.4.0-91-generic #102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
lxc@lxc4:~$

4

Can you also show the content of /var/snap/lxd/common/lxd/logs/lxd.log and the output of lxc monitor --pretty while running such a snapshot delete operation?

5

Hello,

Here it is. Thanks in advance.

Blockquote
lxd delete -f [container_name]
Error: Error deleting storage volume: Cannot remove a volume that has snapshots

$ lxc monitor --pretty
DBUG[12-02|15:57:58] New event listener: 2f996dee-b583-4547-bf5b-28e10995eb05
DBUG[12-02|15:58:20] Handling API request protocol=unix url=/1.0 username=lxc ip=@ method=GET
DBUG[12-02|15:58:20] Handling API request protocol=unix url=“/1.0/instances?recursion=2” username=lxc ip=@ method=GET
DBUG[12-02|15:58:20] GetInstanceUsage started pool=storpool1 project=default driver=btrfs instance=POLOVM153147
DBUG[12-02|15:58:20] GetInstanceUsage started pool=storpool1 project=default driver=btrfs instance=POLOVM153145
DBUG[12-02|15:58:20] GetInstanceUsage started project=default driver=btrfs instance=POLOVM153146 pool=storpool1
DBUG[12-02|15:58:20] GetInstanceUsage started driver=btrfs instance=rintis-qa-db pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage finished driver=btrfs instance=POLOVM153147 pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage finished driver=btrfs instance=POLOVM153146 pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage finished project=default driver=btrfs instance=POLOVM153145 pool=storpool1
DBUG[12-02|15:58:20] GetInstanceUsage finished driver=btrfs instance=rintis-qa-db pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage started driver=btrfs instance=wireguard-extlab852 pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage started driver=btrfs instance=contdesktop pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage started pool=storpool1 project=default driver=btrfs instance=rkelauncher
DBUG[12-02|15:58:20] GetInstanceUsage started driver=btrfs instance=rintislb pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage finished driver=btrfs instance=wireguard-extlab852 pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage finished pool=storpool1 project=default driver=btrfs instance=contdesktop
DBUG[12-02|15:58:20] GetInstanceUsage finished instance=rkelauncher pool=storpool1 project=default driver=btrfs
DBUG[12-02|15:58:20] GetInstanceUsage finished driver=btrfs instance=rintislb pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage started driver=btrfs instance=rintisbdd pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage started driver=btrfs instance=gcpilot pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage finished project=default driver=btrfs instance=rintisbdd pool=storpool1
DBUG[12-02|15:58:20] GetInstanceUsage started driver=btrfs instance=rintis-db-node1 pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage finished driver=btrfs instance=gcpilot pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage finished driver=btrfs instance=rintis-db-node1 pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage started driver=btrfs instance=rintis-db-node2 pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage finished instance=rintis-db-node2 pool=storpool1 project=default driver=btrfs
DBUG[12-02|15:58:20] GetInstanceUsage started driver=btrfs instance=rintis-db-node4 pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage started driver=btrfs instance=rintis-db-node3 pool=storpool1 project=default
DBUG[12-02|15:58:20] GetInstanceUsage finished project=default driver=btrfs instance=rintis-db-node4 pool=storpool1
DBUG[12-02|15:58:20] GetInstanceUsage started project=default driver=btrfs instance=rintis-db-node5 pool=storpool1
DBUG[12-02|15:58:21] GetInstanceUsage started driver=btrfs instance=rintis-db-node6 pool=storpool1 project=default
DBUG[12-02|15:58:21] GetInstanceUsage finished driver=btrfs instance=rintis-db-node3 pool=storpool1 project=default
DBUG[12-02|15:58:21] GetInstanceUsage finished driver=btrfs instance=rintis-db-node5 pool=storpool1 project=default
DBUG[12-02|15:58:21] GetInstanceUsage finished driver=btrfs instance=rintis-db-node6 pool=storpool1 project=default
DBUG[12-02|15:58:26] Handling API request method=GET protocol=unix url=/1.0 username=lxc ip=@
DBUG[12-02|15:58:26] Handling API request ip=@ method=GET protocol=unix url=/1.0/instances/rintis-db-node1 username=lxc
DBUG[12-02|15:58:26] Handling API request method=GET protocol=unix url=/1.0/events username=lxc ip=@
DBUG[12-02|15:58:26] New event listener: 190c1418-a090-4668-bdd9-24a11fd397e2
DBUG[12-02|15:58:26] Handling API request url=/1.0/instances/rintis-db-node1 username=lxc ip=@ method=DELETE protocol=unix
DBUG[12-02|15:58:26] New task Operation: c7fb5ea2-9aa6-4b0e-813b-442674d1bbfb
INFO[12-02|15:58:26] Deleting container created=“2021-12-02 07:13:12.504109864 +0000 UTC” ephemeral=false instance=rintis-db-node1 instanceType=container project=default used=“2021-12-02 08:56:08.259108566 +0000 UTC”
INFO[12-02|15:58:26] ID: c7fb5ea2-9aa6-4b0e-813b-442674d1bbfb, Class: task, Description: Deleting instance CreatedAt=2021-12-02T15:58:26+0000 UpdatedAt=2021-12-02T15:58:26+0000 Status=Pending StatusCode=Pending Resources=“map[containers:[/1.0/containers/rintis-db-node1] instances:[/1.0/instances/rintis-db-node1]]” Metadata=map MayCancel=false Err= Location=none
DBUG[12-02|15:58:26] Started task operation: c7fb5ea2-9aa6-4b0e-813b-442674d1bbfb
INFO[12-02|15:58:26] ID: c7fb5ea2-9aa6-4b0e-813b-442674d1bbfb, Class: task, Description: Deleting instance CreatedAt=2021-12-02T15:58:26+0000 UpdatedAt=2021-12-02T15:58:26+0000 Status=Running StatusCode=Running Resources=“map[containers:[/1.0/containers/rintis-db-node1] instances:[/1.0/instances/rintis-db-node1]]” Metadata=map MayCancel=false Err= Location=none
DBUG[12-02|15:58:26] Handling API request protocol=unix url=/1.0/operations/c7fb5ea2-9aa6-4b0e-813b-442674d1bbfb username=lxc ip=@ method=GET
INFO[12-02|15:58:26] Deleting container instance=rintis-db-node1/6node-drop17.5-dr instanceType=container project=default used=“0001-01-01 00:00:00 +0000 UTC” created=“2021-12-02 08:55:22.799444008 +0000 UTC” ephemeral=false
DBUG[12-02|15:58:26] DeleteInstanceSnapshot started driver=btrfs instance=rintis-db-node1/6node-drop17.5-dr pool=storpool1 project=default
DBUG[12-02|15:58:26] Deleting instance snapshot volume driver=btrfs instance=rintis-db-node1/6node-drop17.5-dr pool=storpool1 project=default snapshotName=6node-drop17.5-dr volName=rintis-db-node1
DBUG[12-02|15:58:32] DeleteInstanceSnapshot finished driver=btrfs instance=rintis-db-node1/6node-drop17.5-dr pool=storpool1 project=default
DBUG[12-02|15:58:32] UpdateInstanceBackupFile started driver=btrfs instance=rintis-db-node1 pool=storpool1 project=default
DBUG[12-02|15:58:32] UpdateInstanceBackupFile finished driver=btrfs instance=rintis-db-node1 pool=storpool1 project=default
INFO[12-02|15:58:32] Action: instance-snapshot-deleted, Source: /1.0/instances/rintis-db-node1/snapshots/6node-drop17.5-dr
INFO[12-02|15:58:32] Deleted container instanceType=container project=default used=“0001-01-01 00:00:00 +0000 UTC” created=“2021-12-02 08:55:22.799444008 +0000 UTC” ephemeral=false instance=rintis-db-node1/6node-drop17.5-dr
DBUG[12-02|15:58:32] DeleteInstance started pool=storpool1 project=default driver=btrfs instance=rintis-db-node1
DBUG[12-02|15:58:32] Deleting instance volume driver=btrfs instance=rintis-db-node1 pool=storpool1 project=default volName=rintis-db-node1
DBUG[12-02|15:58:32] Failure for task operation: c7fb5ea2-9aa6-4b0e-813b-442674d1bbfb: Error deleting storage volume: Cannot remove a volume that has snapshots
DBUG[12-02|15:58:32] DeleteInstance finished driver=btrfs instance=rintis-db-node1 pool=storpool1 project=default
INFO[12-02|15:58:32] ID: c7fb5ea2-9aa6-4b0e-813b-442674d1bbfb, Class: task, Description: Deleting instance CreatedAt=2021-12-02T15:58:26+0000 UpdatedAt=2021-12-02T15:58:26+0000 Status=Failure StatusCode=Failure Resources=“map[containers:[/1.0/containers/rintis-db-node1] instances:[/1.0/instances/rintis-db-node1]]” Metadata=map MayCancel=false Err=“Error deleting storage volume: Cannot remove a volume that has snapshots” Location=none
DBUG[12-02|15:58:32] Event listener finished: 190c1418-a090-4668-bdd9-24a11fd397e2
DBUG[12-02|15:58:32] Disconnected event listener listener=190c1418-a090-4668-bdd9-24a11fd397e2

Blockquote
$ sudo tail -f /var/snap/lxd/common/lxd/logs/lxd.log
[sudo] password for lxc:
t=2021-12-02T15:45:44+0000 lvl=info msg=“Done pruning expired instance backups”
t=2021-12-02T15:45:44+0000 lvl=info msg=“Updating images”
t=2021-12-02T15:45:44+0000 lvl=info msg=“Done updating images”
t=2021-12-02T15:45:44+0000 lvl=info msg=“Pruning resolved warnings”
t=2021-12-02T15:45:44+0000 lvl=info msg=“Expiring log files”
t=2021-12-02T15:45:44+0000 lvl=info msg=“Done pruning resolved warnings”
t=2021-12-02T15:45:44+0000 lvl=info msg=“Done expiring log files”
t=2021-12-02T15:45:44+0000 lvl=info msg=“Updating instance types”
t=2021-12-02T15:45:44+0000 lvl=info msg=“Done updating instance types”
t=2021-12-02T15:45:44+0000 lvl=info msg=“Daemon started”
t=2021-12-02T15:58:26+0000 lvl=info msg=“Deleting container” created=2021-12-02T07:13:12+0000 ephemeral=false instance=rintis-db-node1 instanceType=container project=default used=2021-12-02T08:56:08+0000
t=2021-12-02T15:58:26+0000 lvl=info msg=“Deleting container” created=2021-12-02T08:55:22+0000 ephemeral=false instance=rintis-db-node1/6node-drop17.5-dr instanceType=container project=default used=0001-01-01T00:00:00+0000
t=2021-12-02T15:58:32+0000 lvl=info msg=“Deleted container” created=2021-12-02T08:55:22+0000 ephemeral=false instance=rintis-db-node1/6node-drop17.5-dr instanceType=container project=default used=0001-01-01T00:00:00+0000

6

Ok, so that error is normal and correct, what I’m more interested in is the log for what happened when that initial snapshot was supposed to be deleted but then wasn’t deleted from disk.

7

Hi, Sorry for very late reply. I thought I fixed it already but it seems like it reoccured for other container could be attributed to a very slow disk (backend from iscsi).

t=2022-02-07T03:09:49+0000 lvl=eror msg="DeleteSnapshots: Failed to delete the snapshot" err="Failed setting subvolume writable \"/var/snap/lxd/common/lxd/storage-pools/storpool1/containers-snapshots/POLOVM152144/drop17.5/rootfs/var/lib/docker/btrfs/subvolumes/ff396de34258e04924192eeef19bd6e8c0c61cb952ef7059d2b7d7a114a07191\": Failed to run: btrfs property set -ts /var/snap/lxd/common/lxd/storage-pools/storpool1/containers-snapshots/POLOVM152144/drop17.5/rootfs/var/lib/docker/btrfs/subvolumes/ff396de34258e04924192eeef19bd6e8c0c61cb952ef7059d2b7d7a114a07191 ro false: ERROR: Could not open: No such file or directory" instance=POLOVM152144 project=default snapshot=POLOVM152144/drop17.5
t=2022-02-07T03:10:06+0000 lvl=info msg="Deleted container" created=2021-11-12T08:18:19+0000 ephemeral=false instance=POLOVM152144/drop17.5 instanceType=container project=default used=0001-01-01T00:00:00+0000
t=2022-02-07T03:10:06+0000 lvl=info msg="Creating container" ephemeral=false instance=POLOVM152144/drop17.5 instanceType=container project=default
t=2022-02-07T03:10:06+0000 lvl=info msg="Created container" ephemeral=false instance=POLOVM152144/drop17.5 instanceType=container project=default
t=2022-02-07T03:11:27+0000 lvl=info msg="Deleting container" created=2022-02-07T02:26:01+0000 ephemeral=false instance=POLOVM152144/drop17.5 instanceType=container project=default used=0001-01-01T00:00:00+0000
t=2022-02-07T03:11:27+0000 lvl=info msg="Deleted container" created=2022-02-07T02:26:01+0000 ephemeral=false instance=POLOVM152144/drop17.5 instanceType=container project=default used=0001-01-01T00:00:00+0000
t=2022-02-07T03:11:34+0000 lvl=info msg="Deleting container" created=2021-11-12T08:20:45+0000 ephemeral=false instance=POLOVM152144 instanceType=container project=default used=1970-01-01T00:00:00+0000
t=2022-02-07T03:21:50+0000 lvl=eror msg="Error getting disk usage" err="Unable to find quota group" instance=POLOVM152144 instanceType=container project=default
t=2022-02-07T03:24:00+0000 lvl=info msg="Deleted container" created=2021-11-12T08:20:45+0000 ephemeral=false instance=POLOVM152144 instanceType=container project=default used=1970-01-01T00:00:00+0000
t=2022-02-07T03:24:36+0000 lvl=info msg="Creating container" ephemeral=false instance=POLOVM152144 instanceType=container project=default
t=2022-02-07T03:24:36+0000 lvl=info msg="Created container" ephemeral=false instance=POLOVM152144 instanceType=container project=default
t=2022-02-07T03:24:36+0000 lvl=info msg="Creating container" ephemeral=false instance=POLOVM152144/drop17.5 instanceType=container project=default
t=2022-02-07T03:24:36+0000 lvl=info msg="Created container" ephemeral=false instance=POLOVM152144/drop17.5 instanceType=container project=default
t=2022-02-07T03:27:58+0000 lvl=info msg="Creating container" ephemeral=false instance=POLOVM152144 instanceType=container project=default
t=2022-02-07T03:27:58+0000 lvl=info msg="Created container" ephemeral=false instance=POLOVM152144 instanceType=container project=default
t=2022-02-07T03:27:58+0000 lvl=info msg="Creating container" ephemeral=false instance=POLOVM152144/drop17.5 instanceType=container project=default
t=2022-02-07T03:27:58+0000 lvl=info msg="Created container" ephemeral=false instance=POLOVM152144/drop17.5 instanceType=container project=default
t=2022-02-07T03:48:21+0000 lvl=info msg="Updating images"
t=2022-02-07T03:48:21+0000 lvl=info msg="Pruning expired instance backups"
t=2022-02-07T03:48:21+0000 lvl=info msg="Done pruning expired instance backups"
t=2022-02-07T03:48:21+0000 lvl=info msg="Done updating images"
t=2022-02-07T03:54:44+0000 lvl=info msg="Creating container" ephemeral=false instance=POLOVM152144 instanceType=container project=default
t=2022-02-07T03:54:46+0000 lvl=info msg="Created container" ephemeral=false instance=POLOVM152144 instanceType=container project=default
t=2022-02-07T03:54:47+0000 lvl=info msg="Creating container" ephemeral=false instance=POLOVM152144/drop17.5 instanceType=container project=default
t=2022-02-07T03:54:47+0000 lvl=info msg="Created container" ephemeral=false instance=POLOVM152144/drop17.5 instanceType=container project=default
t=2022-02-07T03:56:20+0000 lvl=info msg="Creating container" ephemeral=false instance=POLOVM152144 instanceType=container project=default
t=2022-02-07T03:56:20+0000 lvl=info msg="Created container" ephemeral=false instance=POLOVM152144 instanceType=container project=default
t=2022-02-07T03:56:20+0000 lvl=info msg="Creating container" ephemeral=false instance=POLOVM152144/drop17.5 instanceType=container project=default
t=2022-02-07T03:56:20+0000 lvl=info msg="Created container" ephemeral=false instance=POLOVM152144/drop17.5 instanceType=container project=default