Error: Error deleting storage volume from database: Storage volume found on more than one cluster member. Please target a specific member

LXD
1

I’m having trouble deleting, moving a container, or deleting a snapshot. This lxd is not clustered and has no additional projects besides default. I am having this same issue on 2 servers, and recall being able to do this before so I think it’s a relatively new issue.

I’m running lxd version 5.9 on both servers. One server is an ubuntu 20.04, and one is a server recently upgraded to ubuntu 22.04.

If I do this:

lxc rm container-name/snap24

I get error:

Error: Error deleting storage volume from database: Storage volume found on more than one cluster member. Please target a specific member

If I try to do this:

 lxc rm container-name

I get error:

Error: Cannot remove an instance volume that has snapshots

If I do

lxc mv container-name container-name-2

I get

Error: Rename instance: Storage volume found on more than one cluster member. Please target a specific member

On a related note, it seems the snapshots are not expiring:

 snap0  | 2022/10/02 22:21 PDT | 2022/11/01 22:21 PDT | NO

Should have been deleted by now, though on the server I am having the same issues with, the snapshot expiring seems to be working fine.

2

Please can you show for both servers:

  • lxc config show <instance> --expanded
  • lxc storage show <pool>
  • lxc storage volume list <pool>
  • lxd sql global 'select * from storage_volumes'

Thanks

3

Sorry for delay in responding. I have the same issue on two servers.
This has been upgraded to 5.10 already

Here is what one resturns

architecture: x86_64
config:
environment.LANG: en_US.UTF-8
environment.LC_ALL: en_US.UTF-8
environment.LC_COLLATE: POSIX
limits.cpu: “4”
limits.memory: 8GB
raw.lxc: lxc.mount.entry = none dev/shm tmpfs rw,nosuid,nodev,create=dir
snapshots.expiry: 10d
snapshots.schedule: 2 18 */4 * *
volatile.cloud-init.instance-id: 657c32c5-d2a7-49b3-b5c0-d40cf0275f4a
volatile.eth0.hwaddr: 00:16:3e:94:38:b2
volatile.idmap.base: “0”
volatile.idmap.current: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.idmap: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.power: STOPPED
volatile.last_state.ready: “false”
volatile.uuid: b7bb3890-13f9-459a-ad9e-5eaa0a8efcd4
devices:
eth0:
name: eth0
nictype: bridged
parent: lxdbr0
type: nic
proxyssh3022:
connect: tcp:127.0.0.1:22
listen: tcp:140.211.15.32:3022
type: proxy
root:
path: /
pool: default
size: 150GB
type: disk
ephemeral: false
profiles:

  • default
    stateful: false
    description: “”

There are a lot, but I am seeing duplicates:

- /1.0/instances/tracsvn-dev/snapshots/snap0
- /1.0/instances/tracsvn-dev/snapshots/snap0
- /1.0/instances/tracsvn-dev/snapshots/snap1
- /1.0/instances/tracsvn-dev/snapshots/snap1
- /1.0/instances/tracsvn-dev/snapshots/snap10
- /1.0/instances/tracsvn-dev/snapshots/snap10
- /1.0/instances/tracsvn-dev/snapshots/snap11
- /1.0/instances/tracsvn-dev/snapshots/snap11
- /1.0/instances/tracsvn-dev/snapshots/snap12
- /1.0/instances/tracsvn-dev/snapshots/snap12
- /1.0/instances/tracsvn-dev/snapshots/snap13
- /1.0/instances/tracsvn-dev/snapshots/snap13
- /1.0/instances/tracsvn-dev/snapshots/snap14
- /1.0/instances/tracsvn-dev/snapshots/snap14
- /1.0/instances/tracsvn-dev/snapshots/snap15
- /1.0/instances/tracsvn-dev/snapshots/snap15
- /1.0/instances/tracsvn-dev/snapshots/snap16
- /1.0/instances/tracsvn-dev/snapshots/snap16
- /1.0/instances/tracsvn-dev/snapshots/snap17
- /1.0/instances/tracsvn-dev/snapshots/snap17
- /1.0/instances/tracsvn-dev/snapshots/snap18
- /1.0/instances/tracsvn-dev/snapshots/snap18
- /1.0/instances/tracsvn-dev/snapshots/snap19
- /1.0/instances/tracsvn-dev/snapshots/snap19
- /1.0/instances/tracsvn-dev/snapshots/snap2
- /1.0/instances/tracsvn-dev/snapshots/snap2
- /1.0/instances/tracsvn-dev/snapshots/snap20
- /1.0/instances/tracsvn-dev/snapshots/snap20
- /1.0/instances/tracsvn-dev/snapshots/snap21
- /1.0/instances/tracsvn-dev/snapshots/snap21

lxc storage volume list

shows:

 container (snapshot) | tracsvn-dev/snap0                                                |             | filesystem   | 1       |
| container (snapshot) | tracsvn-dev/snap0                                                |             | filesystem   | 1       |
| container (snapshot) | tracsvn-dev/snap1                                                |             | filesystem   | 1       |
| container (snapshot) | tracsvn-dev/snap1                                                |             | filesystem   | 1       |
| container (snapshot) | tracsvn-dev/snap2                                                |             | filesystem   | 1       |


lxd sql global "select * from storage_volumes WHERE name LIKE 'tracsvn-dev/snap%' ORDER BY name LIMIT 10;"

I’m seeing no dupes

+-----+--------------------+-----------------+---------+------+-------------+------------+--------------+----------------------+
| id  |        name        | storage_pool_id | node_id | type | description | project_id | content_type |    creation_date     |
+-----+--------------------+-----------------+---------+------+-------------+------------+--------------+----------------------+
| 374 | tracsvn-dev/snap0  | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 382 | tracsvn-dev/snap1  | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 475 | tracsvn-dev/snap10 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 484 | tracsvn-dev/snap11 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 494 | tracsvn-dev/snap12 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 504 | tracsvn-dev/snap13 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 517 | tracsvn-dev/snap14 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 526 | tracsvn-dev/snap15 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 527 | tracsvn-dev/snap16 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 536 | tracsvn-dev/snap17 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
+-----+--------------------+-----------------+---------+------+-------------+------------+--------------+----------------------+
4

Is this a cluster?

5

Also can you show me “lxc project list” please

6

And finally the output of

lxd sql global "select * from storage_volumes_all"

Thanks

7

Running:

 lxd sql global "select * from storage_volumes_all WHERE name LIKE '%tracsvn-dev/snap%' ORDER BY name LIMIT 10"

seeing dupes:

+-------+--------------------+-----------------+---------+------+-------------+------------+--------------+----------------------+
|  id   |        name        | storage_pool_id | node_id | type | description | project_id | content_type |    creation_date     |
+-------+--------------------+-----------------+---------+------+-------------+------------+--------------+----------------------+
| 374   | tracsvn-dev/snap0  | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 14396 | tracsvn-dev/snap0  | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 382   | tracsvn-dev/snap1  | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 14430 | tracsvn-dev/snap1  | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 475   | tracsvn-dev/snap10 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 14923 | tracsvn-dev/snap10 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 484   | tracsvn-dev/snap11 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 14981 | tracsvn-dev/snap11 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 494   | tracsvn-dev/snap12 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 15031 | tracsvn-dev/snap12 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
+-------+--------------------+-----------------+---------+------+-------------+------------+--------------+----------------------+

running:

lxd sql global "select * from storage_volumes_all WHERE name LIKE '%tracsvn-dev/snap1%' ORDER BY name"

I start seeing some with non-crazy creation dates

|  id   |        name        | storage_pool_id | node_id | type | description | project_id | content_type |         creation_date          |
+-------+--------------------+-----------------+---------+------+-------------+------------+--------------+--------------------------------+
| 382   | tracsvn-dev/snap1  | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z           |
| 14430 | tracsvn-dev/snap1  | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z           |
| 475   | tracsvn-dev/snap10 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z           |
| 14923 | tracsvn-dev/snap10 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z           |
| 484   | tracsvn-dev/snap11 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z           |
| 14981 | tracsvn-dev/snap11 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z           |
| 494   | tracsvn-dev/snap12 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z           |
| 15031 | tracsvn-dev/snap12 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z           |
| 504   | tracsvn-dev/snap13 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z           |
| 15091 | tracsvn-dev/snap13 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z           |
| 517   | tracsvn-dev/snap14 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z           |
| 15147 | tracsvn-dev/snap14 | 1               | 1       | 0    |             | 1          | 0            | 2022-11-26T02:01:35.078400865Z |
| 526   | tracsvn-dev/snap15 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z           |
| 15204 | tracsvn-dev/snap15 | 1               | 1       | 0    |             | 1          | 0            | 2022-11-30T02:01:07.282444301Z |
| 527   | tracsvn-dev/snap16 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z           |
| 15233 | tracsvn-dev/snap16 | 1               | 1       | 0    |             | 1          | 0            | 2022-12-02T02:01:50.381070563Z |
| 536   | tracsvn-dev/snap17 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z           |
| 15295 | tracsvn-dev/snap17 | 1               | 1       | 0    |             | 1          | 0            | 2022-12-06T02:01:13.195890766Z |
| 545   | tracsvn-dev/snap18 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z           |
| 15353 | tracsvn-dev/snap18 | 1               | 1       | 0    |             | 1          | 0            | 2022-12-10T02:01:39.091491887Z |
| 554   | tracsvn-dev/snap19 | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z           |
| 15413 | tracsvn-dev/snap19 | 1               | 1       | 0    |             | 1          | 0            | 2022-12-14T02:01:41.055669054Z |
+-------+--------------------+-----------------+---------+------+-------------+------------+--------------+--------------------------------+

For the other server in question that has same issue, both of these were upgraded to Ubuntu 22 from 20 and are now running 5.10

For the other server in question

lxc config show wordpress --expanded

architecture: x86_64
config:
  environment.LANG: en_US.UTF-8
  environment.LC_ALL: en_US.UTF-8
  environment.LC_COLLATE: POSIX
  limits.cpu: "4"
  limits.cpu.allowance: 20%
  limits.memory: 8GB
  raw.lxc: lxc.mount.entry = none dev/shm tmpfs rw,nosuid,nodev,create=dir
  snapshots.expiry: 30d
  snapshots.schedule: 15 18 * * *
  volatile.eth0.hwaddr: 00:16:3e:61:d7:92
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: STOPPED
  volatile.uuid: 5a204571-166d-49dc-90d9-e2cb6e9c536b
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    size: 200GB
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

lxc storage show default | grep wordpress/

- /1.0/instances/wordpress/snapshots/after-2018-foss4g-moved
- /1.0/instances/wordpress/snapshots/after-2018-foss4g-moved
- /1.0/instances/wordpress/snapshots/after-change-to-dhcp
- /1.0/instances/wordpress/snapshots/after-change-to-dhcp
- /1.0/instances/wordpress/snapshots/after-imagemagick-bcmath-install
- /1.0/instances/wordpress/snapshots/after-imagemagick-bcmath-install
- /1.0/instances/wordpress/snapshots/after-moving-www-over
- /1.0/instances/wordpress/snapshots/after-moving-www-over
- /1.0/instances/wordpress/snapshots/bash

Note this one I have stopped a while ago, so all the snapshots that were set to expire have since disappeared.

 lxc storage volume list default | grep wordpress/
| container (snapshot) | wordpress/after-2018-foss4g-moved                                |               | filesystem   | 1       |
| container (snapshot) | wordpress/after-2018-foss4g-moved                                | Auto repaired | filesystem   | 1       |
| container (snapshot) | wordpress/after-change-to-dhcp                                   |               | filesystem   | 1       |
| container (snapshot) | wordpress/after-change-to-dhcp                                   | Auto repaired | filesystem   | 1       |
| container (snapshot) | wordpress/after-imagemagick-bcmath-install                       |               | filesystem   | 1       |
| container (snapshot) | wordpress/after-imagemagick-bcmath-install                       | Auto repaired | filesystem   | 1       |
| container (snapshot) | wordpress/after-moving-www-over                                  |               | filesystem   | 1       |
| container (snapshot) | wordpress/after-moving-www-over                                  | Auto repaired | filesystem   | 1       |
| container (snapshot) | wordpress/bash                                                   |               | filesystem   | 1       |

lxd sql global "select * from storage_volumes WHERE name LIKE 'wordpress/%' ORDER BY name;"

+------+--------------------------------------------+-----------------+---------+------+-------------+------------+--------------+----------------------+
|  id  |                    name                    | storage_pool_id | node_id | type | description | project_id | content_type |    creation_date     |
+------+--------------------------------------------+-----------------+---------+------+-------------+------------+--------------+----------------------+
| 1099 | wordpress/after-2018-foss4g-moved          | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 1097 | wordpress/after-change-to-dhcp             | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 1104 | wordpress/after-imagemagick-bcmath-install | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
| 1100 | wordpress/after-moving-www-over            | 1               | 1       | 0    |             | 1          | 0            | 0001-01-01T00:00:00Z |
+------+--------------------------------------------+-----------------+---------+------+-------------+------------+--------------+----------------------+


lxd sql global "select * from storage_volumes_all WHERE name LIKE 'wordpress/%' ORDER BY name;"

+-------+--------------------------------------------+-----------------+---------+------+---------------+------------+--------------+-------------------------------+
|  id   |                    name                    | storage_pool_id | node_id | type |  description  | project_id | content_type |         creation_date         |
+-------+--------------------------------------------+-----------------+---------+------+---------------+------------+--------------+-------------------------------+
| 1099  | wordpress/after-2018-foss4g-moved          | 1               | 1       | 0    |               | 1          | 0            | 0001-01-01T00:00:00Z          |
| 21827 | wordpress/after-2018-foss4g-moved          | 1               | 1       | 0    | Auto repaired | 1          | 0            | 0001-01-01T00:00:00Z          |
| 1097  | wordpress/after-change-to-dhcp             | 1               | 1       | 0    |               | 1          | 0            | 0001-01-01T00:00:00Z          |
| 21826 | wordpress/after-change-to-dhcp             | 1               | 1       | 0    | Auto repaired | 1          | 0            | 0001-01-01T00:00:00Z          |
| 1104  | wordpress/after-imagemagick-bcmath-install | 1               | 1       | 0    |               | 1          | 0            | 0001-01-01T00:00:00Z          |
| 21829 | wordpress/after-imagemagick-bcmath-install | 1               | 1       | 0    | Auto repaired | 1          | 0            | 0001-01-01T00:00:00Z          |
| 1100  | wordpress/after-moving-www-over            | 1               | 1       | 0    |               | 1          | 0            | 0001-01-01T00:00:00Z          |
| 21828 | wordpress/after-moving-www-over            | 1               | 1       | 0    | Auto repaired | 1          | 0            | 0001-01-01T00:00:00Z          |
| 26602 | wordpress/bash                             | 1               | 1       | 0    |               | 1          | 0            | 2023-02-17T16:40:11.86741912Z |
+-------+--------------------------------------------+-----------------+---------+------+---------------+------------+--------------+-------------------------------+
8

No it’s not a cluster

lxc project list

shows

+-------------------+--------+----------+-----------------+-----------------+----------+---------------+---------------------+---------+
|       NAME        | IMAGES | PROFILES | STORAGE VOLUMES | STORAGE BUCKETS | NETWORKS | NETWORK ZONES |     DESCRIPTION     | USED BY |
+-------------------+--------+----------+-----------------+-----------------+----------+---------------+---------------------+---------+
| default (current) | YES    | YES      | YES             | YES             | YES      | YES           | Default LXD project | 58      |
+-------------------+--------+----------+-----------------+-----------------+----------+---------------+---------------------+---------+

I’ll check the other server with same issue in a minute

9

Right I think I see whats going on here now.

I suspect the valid snapshots are going to be in:

lxd sql global 'select storage_volumes.name, storage_volumes_snapshots.* from storage_volumes_snapshots join storage_volumes on storage_volumes.id = storage_volumes_snapshots.storage_volume_id'

And then the rogue left over ones will be in:

lxd sql global 'select * from storage_volumes where name like "%/%"'

It looks like you’re affected by https://github.com/lxc/lxd/issues/11292

10

You should be able to do:

lxd sql global 'delete from storage_volumes where id = <ID>'

For the rogue entries in storage_volumes.

11

Thanks that worked. I seemed to have very many of these across several containers, too many to delete one at a time.

To fix I did this:

To find them I used this query

-- verify the names look right
lxd sql global "SELECT storage_volumes.id, storage_volumes.name FROM storage_volumes LEFT JOIN storage_volumes_snapshots on storage_volumes.id = storage_volumes_snapshots.storage_volume_id
WHERE storage_volumes_snapshots.storage_volume_id IS NULL AND storage_volumes.name LIKE '%';"

– get list of ids

lxd sql global "select group_concat(storage_volumes.id, ',') from storage_volumes left join storage_volumes_snapshots on storage_volumes.id = storage_volumes_snapshots.storage_volume_id
WHERE storage_volumes_snapshots.storage_volume_id IS NULL  AND storage_volumes.name LIKE '%';"

Delete the orphaned volumes

lxd sql global 'DELETE FROM storage_volumes WHERE id IN(<list of ids from above query>)'
1 Like
12

Sorry I didn’t understand why you were doing the ‘%/%’ So my query should have been below
Without that, you end up deleting volumes of instances that have no snapshots.

-- verify the names look right
lxd sql global "SELECT storage_volumes.id, storage_volumes.name FROM storage_volumes LEFT JOIN storage_volumes_snapshots on storage_volumes.id = storage_volumes_snapshots.storage_volume_id
WHERE storage_volumes_snapshots.storage_volume_id IS NULL AND storage_volumes.name LIKE '%/%';"

lxd sql global "select group_concat(storage_volumes.id, ',') from storage_volumes left join storage_volumes_snapshots on storage_volumes.id = storage_volumes_snapshots.storage_volume_id
WHERE storage_volumes_snapshots.storage_volume_id IS NULL  AND storage_volumes.name LIKE '%/%';"