Cannot remove cluster member because "Image not found"

LXD
1

The cluster has six members.
I want to remove one cluster member.
I have moved all instances to other cluster members.
I ran lxc cluster evacuate memberX

Then I wanted to remove memberX from the cluster. It failed with

root@ijssel:~# lxc cluster remove luts
Error: Failed getting image info for "39a93d0b355279d430e8ce21c689aa88515212ee99874276e77f7f31ad7bf810": Image not found

root@ijssel:~# lxc cluster ls
+---------+---------------------------+------------------+--------------+----------------+-------------+-----------+--------------------------------+
|  NAME   |            URL            |      ROLES       | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION |   STATE   |            MESSAGE             |
+---------+---------------------------+------------------+--------------+----------------+-------------+-----------+--------------------------------+
| angstel | https://172.16.16.76:8443 | database-standby | x86_64       | default        |             | ONLINE    | Fully operational              |
+---------+---------------------------+------------------+--------------+----------------+-------------+-----------+--------------------------------+
| ijssel  | https://172.16.16.54:8443 | database-standby | x86_64       | default        |             | ONLINE    | Fully operational              |
+---------+---------------------------+------------------+--------------+----------------+-------------+-----------+--------------------------------+
| luts    | https://172.16.16.45:8443 |                  | x86_64       | default        |             | EVACUATED | Unavailable due to maintenance |
+---------+---------------------------+------------------+--------------+----------------+-------------+-----------+--------------------------------+
| maas    | https://172.16.16.20:8443 | database         | x86_64       | default        |             | ONLINE    | Fully operational              |
+---------+---------------------------+------------------+--------------+----------------+-------------+-----------+--------------------------------+
| rijn    | https://172.16.16.59:8443 | database         | x86_64       | default        |             | ONLINE    | Fully operational              |
+---------+---------------------------+------------------+--------------+----------------+-------------+-----------+--------------------------------+
| roer    | https://172.16.16.33:8443 | database-leader  | x86_64       | default        |             | ONLINE    | Fully operational              |
|         |                           | database         |              |                |             |           |                                |
+---------+---------------------------+------------------+--------------+----------------+-------------+-----------+--------------------------------+
2

It’s unclear to me what image that is. It is not in the images list. It is also not in the list when doing lxc storage volume ls local (my storage pool is called local)

Here is the command that I used to list “image” storage volumes on that cluster member (called luts). Again, there is no image 39a93d0b3552.

root@roer:~# lxc storage volume ls local -cnLt -fcsv | grep -v container | grep -v virtual-machine | awk -F, '$2 == "luts"{printf("%s  %s\n", $1, $2)}'
be72d4c41e06d2f959a2c1fc761fa01b4b4aad7f2d8e9870ba4114eede7fd2a6  luts
c14665acda2842525ec9b080baba17c3ac43225fc045ea1e52df939562c84b31  luts
e299296138c256b79dda4e61ac7454cf4ac134b43f5521f1ac894f49a9421d00  luts

However, there is a LVM volume with the fingerprint from the error message.

root@luts:~# lvs local
  LV                                                                      VG    Attr       LSize  Pool        Origin Data%  Meta%  Move Log Cpy%Sync Convert
  LXDThinPool                                                             local twi-aotz--  1.59t                    0.23   1.97                            
  images_39a93d0b355279d430e8ce21c689aa88515212ee99874276e77f7f31ad7bf810 local Vwi---tz-k <9.32g LXDThinPool                                               
  images_be72d4c41e06d2f959a2c1fc761fa01b4b4aad7f2d8e9870ba4114eede7fd2a6 local Vwi---tz-k 10.00g LXDThinPool                                               
  images_c14665acda2842525ec9b080baba17c3ac43225fc045ea1e52df939562c84b31 local Vwi---tz-k 10.00g LXDThinPool                                               
  images_e299296138c256b79dda4e61ac7454cf4ac134b43f5521f1ac894f49a9421d00 local Vwi---tz-k 10.00g LXDThinPool

It looks like LXC is confused about some images. It thinks it’s there and it also it cannot find it.

3

Maybe try with --force?

This definitely sounds like a bug, but that’d be something to bring up on the other forum.

4

Where do you want to continue the discussion. I have some more details to share.

5

Since this is a LXD issue and not an Incus one, it would have to go on Canonical’s forum and through the LXD team.

6

How bad is it to --force delete a cluster member? What garbage does that leave behind?

7

It won’t leave anything behind on the server but may not fully reset the machine being removed.

1 Like
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.