Hello,
I’m trying to remove a node from the cluster but I can’t seem to get it to go.
I keep getting this error
Error: Error deleting image "ef5e9bfe1a9c1d5018b6560c39b0e21cf18048b73959d7d88c896ef64513e661" from storage pool "local": Get image "ef5e9bfe1a9c1d5018b6560c39b0e21cf18048b73959d7d88c896ef64513e661": Image not found
I went on the actual host and delete, or at least I think, all the images.
Thank you,
lxd sql global "SELECT * FROM nodes"lxd sql global "SELECT * FROM images"lxd sql global "SELECT * FROM images_nodes"That should help get a clear picture of what’s going on.
Hey @stgraber… Here is the output all of those commands. I still don’t know whats going on 
lxd sql global "SELECT * FROM nodes"
+----+--------+-------------+--------------------+--------+----------------+--------------------------------+-------+------+-------------------+
| id | name | description | address | schema | api_extensions | heartbeat | state | arch | failure_domain_id |
+----+--------+-------------+--------------------+--------+----------------+--------------------------------+-------+------+-------------------+
| 1 | raijin | | 192.168.20.50:8443 | 66 | 327 | 2022-11-12T17:05:27.469972361Z | 0 | 2 | <nil> |
| 2 | anuke | | 192.168.20.51:8443 | 66 | 327 | 2022-11-12T17:05:24.068817493Z | 2 | 2 | <nil> |
| 3 | nox | | 192.168.20.52:8443 | 66 | 327 | 2022-11-12T17:05:25.347869955Z | 0 | 2 | <nil> |
| 4 | baku | | 192.168.20.53:8443 | 66 | 327 | 2022-11-12T17:05:26.395574589Z | 0 | 2 | <nil> |
+----+--------+-------------+--------------------+--------+----------------+--------------------------------+-------+------+-------------------+
adrean@nox:~$ lxd sql global "SELECT * FROM images"
+----+-------------+----------+------+--------+--------------+---------------+-------------+-------------+--------+---------------+-------------+------------+------+
| id | fingerprint | filename | size | public | architecture | creation_date | expiry_date | upload_date | cached | last_use_date | auto_update | project_id | type |
+----+-------------+----------+------+--------+--------------+---------------+-------------+-------------+--------+---------------+-------------+------------+------+
+----+-------------+----------+------+--------+--------------+---------------+-------------+-------------+--------+---------------+-------------+------------+------+
adrean@nox:~$ lxd sql global "SELECT * FROM images_nodes"
+----+----------+---------+
| id | image_id | node_id |
+----+----------+---------+
+----+----------+---------+
What version of LXD is this?
Looks like you have an orphaned image volume on your storage pool.
What does lxc storage volume list <pool> and lxc storage show <pool> show?
lxc storage volume list local
+-----------+------------------------------------------------------------------+-------------+--------------+---------+----------+
| TYPE | NAME | DESCRIPTION | CONTENT-TYPE | USED BY | LOCATION |
+-----------+------------------------------------------------------------------+-------------+--------------+---------+----------+
| container | nds | | filesystem | 1 | nox |
+-----------+------------------------------------------------------------------+-------------+--------------+---------+----------+
| container | pi-hole | | filesystem | 1 | raijin |
+-----------+------------------------------------------------------------------+-------------+--------------+---------+----------+
| container | pi-hole2 | | filesystem | 1 | baku |
+-----------+------------------------------------------------------------------+-------------+--------------+---------+----------+
| container | tech-dns | | filesystem | 1 | baku |
+-----------+------------------------------------------------------------------+-------------+--------------+---------+----------+
| image | 990536288cd0ba179afdf99dfb68721e264a0a344ba2426bcba4a6da64895453 | | filesystem | 1 | baku |
+-----------+------------------------------------------------------------------+-------------+--------------+---------+----------+
and
lxc storage show local
config: {}
description: ""
name: local
driver: zfs
used_by:
- /1.0/images/990536288cd0ba179afdf99dfb68721e264a0a344ba2426bcba4a6da64895453?target=baku
- /1.0/instances/grafana?project=internal-services
- /1.0/instances/nds
- /1.0/instances/nzbget?project=media-services
- /1.0/instances/pi-hole
- /1.0/instances/pi-hole2
- /1.0/instances/plex?project=media-services
- /1.0/instances/prometheus?project=internal-services
- /1.0/instances/prowlarr?project=media-services
- /1.0/instances/radarr?project=media-services
- /1.0/instances/sonarr?project=media-services
- /1.0/instances/tech-dns
- /1.0/instances/uptime-kuma?project=internal-services
- /1.0/instances/vpn-server?project=external-services
- /1.0/profiles/default
- /1.0/storage-pools/local/volumes/image/2d673c8f8755651deaea70a6973335ba2d9239695d41878c1ffd1def30e8dceb?target=nox
- /1.0/storage-pools/local/volumes/image/6d15e60d2685df886f2030244070aa6f84d26739474f2f03eb2cf541e9424cc4?target=nox
- /1.0/storage-pools/local/volumes/image/dcdf86f2a6d848c6e46777835be6ea15f2950bc7905ca326687b52c99e2fb510?target=nox
- /1.0/storage-pools/local/volumes/image/ef5e9bfe1a9c1d5018b6560c39b0e21cf18048b73959d7d88c896ef64513e661?target=anuke
status: Created
locations:
- nox
- baku
- raijin
- anuke
Can you run lxc storage volume list local on cluster member anuke please.
It looks like you have a left over storage volume.
Can you also show sudo zfs list on anuke please.
I’ve created Storage: Update DeleteImage to not use GetImageFromAnyProject by tomponline · Pull Request #11118 · lxc/lxd · GitHub which should also work around such a problem in the future.
Here is the results of the command!
NAME USED AVAIL REFER MOUNTPOINT
local 210M 89.9G 24K legacy
local/buckets 24K 89.9G 24K legacy
local/containers 24K 89.9G 24K legacy
local/custom 24K 89.9G 24K legacy
local/deleted 144K 89.9G 24K legacy
local/deleted/buckets 24K 89.9G 24K legacy
local/deleted/containers 24K 89.9G 24K legacy
local/deleted/custom 24K 89.9G 24K legacy
local/deleted/images 24K 89.9G 24K legacy
local/deleted/virtual-machines 24K 89.9G 24K legacy
local/images 186M 89.9G 24K legacy
local/images/ef5e9bfe1a9c1d5018b6560c39b0e21cf18048b73959d7d88c896ef64513e661 186M 89.9G 186M legacy
local/virtual-machines 24K 89.9G 24K legacy
Also see above:
So sorry! Missed that Here you are from anuke.
lxc storage volume list local
+-----------+------------------------------------------------------------------+-------------+--------------+---------+----------+
| TYPE | NAME | DESCRIPTION | CONTENT-TYPE | USED BY | LOCATION |
+-----------+------------------------------------------------------------------+-------------+--------------+---------+----------+
| container | nds | | filesystem | 1 | nox |
+-----------+------------------------------------------------------------------+-------------+--------------+---------+----------+
| container | pi-hole | | filesystem | 1 | raijin |
+-----------+------------------------------------------------------------------+-------------+--------------+---------+----------+
| container | pi-hole2 | | filesystem | 1 | raijin |
+-----------+------------------------------------------------------------------+-------------+--------------+---------+----------+
| container | tech-dns | | filesystem | 1 | baku |
+-----------+------------------------------------------------------------------+-------------+--------------+---------+----------+
| image | 990536288cd0ba179afdf99dfb68721e264a0a344ba2426bcba4a6da64895453 | | filesystem | 1 | baku |
+-----------+------------------------------------------------------------------+-------------+--------------+---------+----------+
I might of jumped the gun but I delete the volume from zfs to clear anything else that I don’t think should be there.
However, now I try to remove the node and get…
lxc cluster rm anuke
Error: Peer not found
lxc cluster list
+--------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE | MESSAGE |
+--------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| anuke | https://192.168.20.51:8443 | database-standby | x86_64 | default | | ONLINE | Fully operational |
+--------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| baku | https://192.168.20.53:8443 | database-leader | x86_64 | default | | ONLINE | Fully operational |
| | | database | | | | | |
+--------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| nox | https://192.168.20.52:8443 | database | x86_64 | default | | ONLINE | Fully operational |
+--------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| raijin | https://192.168.20.50:8443 | database | x86_64 | default | | ONLINE | Fully operational |
+--------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
Nevermind! I restarted the lxd daemon and then got the same error as before.
Error: Error deleting image "ef5e9bfe1a9c1d5018b6560c39b0e21cf18048b73959d7d88c896ef64513e661" from storage pool "local": Get image "ef5e9bfe1a9c1d5018b6560c39b0e21cf18048b73959d7d88c896ef64513e661": Image not found
Try going “lxc storage volume remove local image/
990536288cd0ba179afdf99dfb68721e264a0a344ba2426bcba4a6da64895453”
And then try again.
Nope still giving me the same error
lxc storage volume list local
+-----------+----------+-------------+--------------+---------+----------+
| TYPE | NAME | DESCRIPTION | CONTENT-TYPE | USED BY | LOCATION |
+-----------+----------+-------------+--------------+---------+----------+
| container | nds | | filesystem | 1 | nox |
+-----------+----------+-------------+--------------+---------+----------+
| container | pi-hole | | filesystem | 1 | raijin |
+-----------+----------+-------------+--------------+---------+----------+
| container | pi-hole2 | | filesystem | 1 | raijin |
+-----------+----------+-------------+--------------+---------+----------+
| container | tech-dns | | filesystem | 1 | baku |
+-----------+----------+-------------+--------------+---------+----------+
Here is me trying to remove it from the cluster
lxc cluster rm anuke
Error: Error deleting image "ef5e9bfe1a9c1d5018b6560c39b0e21cf18048b73959d7d88c896ef64513e661" from storage pool "local": Get image "ef5e9bfe1a9c1d5018b6560c39b0e21cf18048b73959d7d88c896ef64513e661": Image not found
Can you show
lxc storage show local
again please
Also zfs list again
Here you go!
lxc storage show local
config: {}
description: ""
name: local
driver: zfs
used_by:
- /1.0/instances/grafana?project=internal-services
- /1.0/instances/nds
- /1.0/instances/nzbget?project=media-services
- /1.0/instances/pi-hole
- /1.0/instances/pi-hole2
- /1.0/instances/plex?project=media-services
- /1.0/instances/prometheus?project=internal-services
- /1.0/instances/prowlarr?project=media-services
- /1.0/instances/radarr?project=media-services
- /1.0/instances/sonarr?project=media-services
- /1.0/instances/tech-dns
- /1.0/instances/uptime-kuma?project=internal-services
- /1.0/instances/vpn-server?project=external-services
- /1.0/profiles/default
- /1.0/storage-pools/local/volumes/image/2d673c8f8755651deaea70a6973335ba2d9239695d41878c1ffd1def30e8dceb?target=nox
- /1.0/storage-pools/local/volumes/image/6d15e60d2685df886f2030244070aa6f84d26739474f2f03eb2cf541e9424cc4?target=nox
- /1.0/storage-pools/local/volumes/image/dcdf86f2a6d848c6e46777835be6ea15f2950bc7905ca326687b52c99e2fb510?target=nox
- /1.0/storage-pools/local/volumes/image/ef5e9bfe1a9c1d5018b6560c39b0e21cf18048b73959d7d88c896ef64513e661?target=anuke
status: Created
locations:
- raijin
- anuke
- nox
- baku
NAME USED AVAIL REFER MOUNTPOINT
local 24.4M 90.1G 24K legacy
local/buckets 24K 90.1G 24K legacy
local/containers 24K 90.1G 24K legacy
local/custom 24K 90.1G 24K legacy
local/deleted 144K 90.1G 24K legacy
local/deleted/buckets 24K 90.1G 24K legacy
local/deleted/containers 24K 90.1G 24K legacy
local/deleted/custom 24K 90.1G 24K legacy
local/deleted/images 24K 90.1G 24K legacy
local/deleted/virtual-machines 24K 90.1G 24K legacy
local/images 24K 90.1G 24K legacy
local/virtual-machines 24K 90.1G 24K legacy