Unable to delete btrfs volume

Hi!

Seems that I had some kernel shenanigans happening with btrfs and created inconsistencies with what LXD is seeing on the database and what is happening on disk.

The issue happened with kernel 5.19.0-43-generic and LXD 5.14-7072c7b

I am trying to delete a btrfs volume and I am getting the following error:

Error: Failed setting subvolume writable "/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker": Failed to run: btrfs property set -f -ts /var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker ro false: exit status 1 (ERROR: Could not get subvolume flags: Invalid argument)

There is nothing inside /var/snap/lxd/common/lxd/storage-pools/docker/:

 $ sudo ls -al /var/snap/lxd/common/lxd/storage-pools/docker/                                                                                                                                                                                                                                                                 2 ↵
total 18
drwx--x--x 2 root root 2 jun 20 08:40 .
drwx--x--x 4 root root 4 jun 20 08:40 ..

But the volume/storage volume does exist in the database:

$ lxc storage volume list docker
+--------+--------+-------------+--------------+---------+
|  TYPE  |  NAME  | DESCRIPTION | CONTENT-TYPE | USED BY |
+--------+--------+-------------+--------------+---------+
| custom | docker |             | filesystem   | 0       |
+--------+--------+-------------+--------------+---------+
$ lxc storage volume show docker docker                                                                                                                                                                                                                                                                                      1 ↵
config:
  volatile.idmap.last: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
description: ""
name: docker
type: custom
used_by: []
location: none
content_type: filesystem
project: default
created_at: 0001-01-01T00:00:00Z
$ lxc storage show docker
config:
  size: 30GiB
  source: /var/snap/lxd/common/lxd/disks/docker.img
description: ""
name: docker
driver: btrfs
used_by:
- /1.0/storage-pools/docker/volumes/custom/docker
status: Created
locations:
- none

And the above does exist:

$ sudo ls -al /var/snap/lxd/common/lxd/disks/docker.img                                                                                                                                                                                                                                                                      2 ↵
-rw------- 1 root root 32212254720 jun 20 09:59 /var/snap/lxd/common/lxd/disks/docker.img

Should I get on with doing database surgery, or any other suggestion?

Thanks in advance.

Are you expecting there to be something in the docker.img pool file?

There should be old data there, but can be deleted.

Does lxc storage list show the pool as unavailable? Perhaps there is an issue mounting the pool file.

Looks normal to me (same as the lxc storage show on the intial post):

+---------+--------+-------------------------------------------+-------------+---------+---------+
| docker  | btrfs  | /var/snap/lxd/common/lxd/disks/docker.img |             | 1       | CREATED |
+---------+--------+-------------------------------------------+-------------+---------+---------+

Ah good, can you check inside the lxd snap’s mount namespace:

sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- ls -al /var/snap/lxd/common/lxd/storage-pools/docker/ 

Ah yes, I can indeed see inside the lxd snap mount namespace:

$ sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- ls -al /var/snap/lxd/common/lxd/storage-pools/docker/                                                                                                                                                                                                                        1 ↵

total 25
drwxr-xr-x 1 root root 200 Jun 20 08:40 .
drwx--x--x 4 root root   4 Jun 20 08:40 ..
drwx--x--x 1 root root   0 Jun 20 08:40 containers
drwx--x--x 1 root root   0 Jun 20 08:40 containers-snapshots
drwx--x--x 1 root root  28 Jun 20 09:20 custom
drwx--x--x 1 root root   0 Jun 20 08:40 custom-snapshots
drwx--x--x 1 root root 128 Jun 20 08:41 images
drwx--x--x 1 root root   0 Jun 20 08:40 virtual-machines
drwx--x--x 1 root root   0 Jun 20 08:40 virtual-machines-snapshots

and the volume:

$ sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- ls -al /var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/

total 4
drwx--x--- 1 1000000 1000000 202 Jun 20 09:45 .
drwx--x--x 1 root    root     28 Jun 20 09:20 ..
drwx--x--x 1 1000000 1000000 152 Jun 20 09:45 buildkit
drwx--x--- 1 1000000 1000000   0 Jun 20 09:22 containers
-rw------- 1 1000000 1000000  36 Jun 20 09:22 engine-id
drwx------ 1 1000000 1000000  16 Jun 20 09:22 image
drwx------ 1 1000000 1000000  28 Jun 20 09:22 metacopy-check3321666892
drwxr-x--- 1 1000000 1000000  10 Jun 20 09:22 network
drwx--x--- 1 1000000 1000000   2 Jun 20 09:45 overlay2
drwx------ 1 1000000 1000000  20 Jun 20 09:22 plugins
drwx------ 1 1000000 1000000   0 Jun 20 09:45 runtimes
drwx------ 1 1000000 1000000   0 Jun 20 09:45 swarm
drwx------ 1 1000000 1000000   0 Jun 20 09:45 tmp
drwx-----x 1 1000000 1000000  22 Jun 20 09:22 volumes
$ sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- ls -al /var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/volumes

total 24
drwx-----x 1 1000000 1000000    22 Jun 20 09:22 .
drwx--x--- 1 1000000 1000000   202 Jun 20 09:45 ..
-rw------- 1 1000000 1000000 32768 Jun 20 09:45 metadata.db

Can you do:

sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- rm -rvf /var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/volumes

heh, so… I made a mistake and things got solved :sweat_smile:

Proceeded to delete:

$ sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- rm -rvf /var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/volumes

removed '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/volumes/metadata.db'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/volumes'

tried lxc storage volume remove docker to no avail.

then instead of doing an ls, I did a rm:

$ sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- rm -rvf /var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker        

removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/containers'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/plugins/tmp'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/plugins/storage/ingest'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/plugins/storage'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/plugins'
removed '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/engine-id'
removed '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/metacopy-check3321666892/l1/f'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/metacopy-check3321666892/l1'
removed '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/metacopy-check3321666892/l2/f'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/metacopy-check3321666892/l2'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/metacopy-check3321666892/work/work'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/metacopy-check3321666892/work'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/metacopy-check3321666892/merged'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/metacopy-check3321666892'
removed '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/image/overlay2/repositories.json'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/image/overlay2/layerdb'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/image/overlay2/imagedb/content/sha256'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/image/overlay2/imagedb/content'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/image/overlay2/imagedb/metadata/sha256'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/image/overlay2/imagedb/metadata'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/image/overlay2/imagedb'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/image/overlay2/distribution'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/image/overlay2'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/image'
removed '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/network/files/local-kv.db'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/network/files'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/network'
removed '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/buildkit/containerdmeta.db'
removed '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/buildkit/snapshots.db'
removed '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/buildkit/metadata_v2.db'
removed '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/buildkit/cache.db'
removed '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/buildkit/history.db'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/buildkit/content/ingest'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/buildkit/content'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/buildkit/executor'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/buildkit'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/tmp'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/runtimes'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/overlay2/l'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/overlay2'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker/swarm'
removed directory '/var/snap/lxd/common/lxd/storage-pools/docker/custom/default_docker'
 $ lxc storage list
+---------+--------+-------------------------------------------+-------------+---------+---------+
|  NAME   | DRIVER |                  SOURCE                   | DESCRIPTION | USED BY |  STATE  |
+---------+--------+-------------------------------------------+-------------+---------+---------+
| default | zfs    | rpool/lxd                                 |             | 19      | CREATED |
+---------+--------+-------------------------------------------+-------------+---------+---------+
| docker  | btrfs  | /var/snap/lxd/common/lxd/disks/docker.img |             | 1       | CREATED |
+---------+--------+-------------------------------------------+-------------+---------+---------+
 $ lxc storage volume delete docker docker                                                                                          
Storage volume docker deleted
 $ lxc storage delete docker
Storage pool docker deleted

so all good now, thanks for the help and pointers :slight_smile:

1 Like