667
July 23, 2019, 3:47pm
1
I’ve deleted the zfs pool manually and still I’m unable to get lxc to remove this storage “lxd” (the name of the zfs pool I created). When I rebooted the server lxd refused to start because the lxd zfs pool was missing. However I can’t figure out where the metadata is coming from and/or how to change the default pool to a different zfs pool.
How do I change the default storage pool to lxdmirror and remove all traces of lxd… and get lxd to read from the right storage pool on start up?
# lxc storage delete lxd
Error: No such Object
# lxc storage list lxd
+-----------+-------------+--------+-----------+---------+
| NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
+-----------+-------------+--------+-----------+---------+
| lxd | | zfs | lxd | 2 |
+-----------+-------------+--------+-----------+---------+
| lxdmirror | | zfs | lxdmirror | 15 |
+-----------+-------------+--------+-----------+---------+
# lxc storage show lxd
config:
source: lxd
zfs.pool_name: lxd
description: ""
name: lxd
driver: zfs
used_by:
- /1.0/images/beb29682c9b598c824592166ee619c8478f5ccba226f525a7d7c0e8810bf3874
- /1.0/images/e1e62217dabb1acff585f13472af44b2720839546d1c3fb60d6187afa91fc995
status: Created
locations:
- none
stgraber
July 24, 2019, 2:23am
2
The errors may be coming from those images.lxc storage volume list lxd and then attempt to delete them with:
lxc storage volume delete lxd image/beb29682c9b598c824592166ee619c8478f5ccba226f525a7d7c0e8810bf3874
lxc storage volume delete lxd image/e1e62217dabb1acff585f13472af44b2720839546d1c3fb60d6187afa91fc995
It could be that this is what’s failing and preventing the deletion of the storage pool.
If they don’t go away willingly (likely since you wiped the pool), you may need a bit of DB surgery, can you show:
lxd sql global “SELECT * FROM storage_pools;”
lxd sql global “SELECT * FROM storage_volumes;”
1 Like
muhfiasbin
September 23, 2020, 4:57am
4
Sorry for bumping this thread again.
Recently I had this problem too.
$ lxc storage volume list local
+------+------+-------------+---------+----------+
| TYPE | NAME | DESCRIPTION | USED BY | LOCATION |
+------+------+-------------+---------+----------+
It show empty list
But, when I try to delete it
$ lxc storage delete local
Error: Failed getting image info for “2cfc5a5567b8d74c0986f3d8a77a2a78e58fe22ea9abd2693112031f85afa1a1”: No such object
I tried the DB surgery command
$ lxd sql global “SELECT * FROM storage_pools;”
+----+--------------+--------+-------------+-------+
| id | name | driver | description | state |
+----+--------------+--------+-------------+-------+
| 1 | local | zfs | | 1 |
| 6 | brickpool | zfs | | 1 |
| 7 | datapool | zfs | | 1 |
| 9 | ujian_online | cephfs | | 1 |
+----+--------------+--------+-------------+-------+
$ lxd sql global “SELECT * FROM storage_volumes WHERE name=‘2cfc5a5567b8d74c0986f3d8a77a2a78e58fe22ea9abd2693112031f85afa1a1’;”
+----+------------------------------------------------------------------+-----------------+---------+------+-------------+------------+
| id | name | storage_pool_id | node_id | type | description | project_id |
+----+------------------------------------------------------------------+-----------------+---------+------+-------------+------------+
| 2 | 2cfc5a5567b8d74c0986f3d8a77a2a78e58fe22ea9abd2693112031f85afa1a1 | 1 | 1 | 1 | | 1 |
| 6 | 2cfc5a5567b8d74c0986f3d8a77a2a78e58fe22ea9abd2693112031f85afa1a1 | 1 | 3 | 1 | | 1 |
+----+------------------------------------------------------------------+-----------------+---------+------+-------------+------------+
How can I get rid this image from the pool?
tomp
September 23, 2020, 8:01am
5
Can you show the output of:
lxd sql global 'select * from images where fingerprint = "2cfc5a5567b8d74c0986f3d8a77a2a78e58fe22ea9abd2693112031f85afa1a1"'
My suspicion is that you won’t get a row back, and this is what is causing the error when deleting the pool here: https://github.com/lxc/lxd/blob/66fd330d466b22ed474717c6985aae632da583fc/lxd/storage_pools.go#L615-L618
@stgraber is there any particular reason why we are checking whether the image exists first before attempting to delete the image volume? As the only property we use from the returned imgInfo is the Fingerprint field, which we already have as the volume name? Is the check there purely to confirm that the volume we are about to delete is indeed an image volume? If so then we could change the query to return the volume type instead and check that.
sava
April 28, 2021, 10:14am
6
Hi,
joostn
May 11, 2022, 2:32pm
7
I’m also running into this. I’m trying to do a rolling OS upgrade by removing nodes from the cluster and adding them back. But:
root@duke:/tmp# lxc cluster remove miles
Error: Failed getting image info for "c1549a94399b666577d113f163194669dfb5e59291e1c35be856e73a56c8b080": Image not found
root@duke:/tmp# lxd sql global "SELECT * FROM storage_pools;"
+----+--------------+--------+-------------------------+-------+
| id | name | driver | description | state |
+----+--------------+--------+-------------------------+-------+
| 1 | cephpool | ceph | Ceph pool for lxd disks | 1 |
| 3 | zfspool | zfs | | 1 |
| 6 | zfsencrypted | zfs | | 1 |
+----+--------------+--------+-------------------------+-------+
root@duke:/tmp# lxd sql global "SELECT * FROM storage_volumes;"
+------+------------------------------------------------------------------+-----------------+---------+------+-------------+------------+--------------+
| id | name | storage_pool_id | node_id | type | description | project_id | content_type |
+------+------------------------------------------------------------------+-----------------+---------+------+-------------+------------+--------------+
| 8 | sensible | 1 | <nil> | 0 | | 1 | 0 |
| 9 | unifi | 1 | <nil> | 0 | | 1 | 0 |
| 15 | minecraft | 1 | <nil> | 0 | | 1 | 0 |
| 34 | homeauto | 1 | <nil> | 0 | | 1 | 0 |
| 37 | fileserver | 1 | <nil> | 0 | | 1 | 0 |
| 107 | win10 | 1 | <nil> | 3 | | 1 | 1 |
| 136 | win11 | 1 | <nil> | 3 | | 1 | 1 |
| 159 | testwebserver | 1 | <nil> | 0 | | 1 | 0 |
| 160 | testactiva | 1 | <nil> | 0 | | 1 | 0 |
| 175 | win11test | 1 | <nil> | 3 | | 1 | 1 |
| 338 | 9933aef762196222cd2a7220ef4f1798fedf6a29dfb1790c32239cf2b888aeea | 3 | 2 | 1 | | 1 | 1 |
| 366 | 6645c75aa225491eea6c83ae7c4b94c9b936604e44828b3330f9b40b29653dc5 | 3 | 1 | 1 | | 1 | 1 |
| 367 | 6645c75aa225491eea6c83ae7c4b94c9b936604e44828b3330f9b40b29653dc5 | 3 | 3 | 1 | | 1 | 1 |
| 527 | dmz-duke | 1 | <nil> | 0 | | 1 | 0 |
| 541 | dmz-miles | 1 | <nil> | 0 | | 1 | 0 |
| 1346 | ubuntupt | 1 | <nil> | 0 | | 1 | 0 |
| 1440 | 51fa4062f683fcd3deeac510811067c91a8fcb4c942bba9a8619bb24be1b175e | 3 | 1 | 1 | | 1 | 0 |
| 1444 | 51fa4062f683fcd3deeac510811067c91a8fcb4c942bba9a8619bb24be1b175e | 3 | 3 | 1 | | 1 | 0 |
| 1663 | testdb3 | 6 | 2 | 0 | | 1 | 0 |
| 1665 | testdb1 | 6 | 3 | 0 | | 1 | 0 |
| 1785 | c1549a94399b666577d113f163194669dfb5e59291e1c35be856e73a56c8b080 | 6 | 3 | 1 | | 1 | 0 |
| 1786 | c1549a94399b666577d113f163194669dfb5e59291e1c35be856e73a56c8b080 | 6 | 1 | 1 | | 1 | 0 |
| 1803 | 3851f02f6276f147caf17ffb23fc993a9c4fd789c7122de05fd4be5980b9de97 | 6 | 2 | 1 | | 1 | 0 |
| 2160 | ubuntu22install | 6 | 3 | 3 | | 1 | 1 |
| 2161 | testdb2 | 6 | 3 | 0 | | 1 | 0 |
+------+------------------------------------------------------------------+-----------------+---------+------+-------------+------------+--------------+
root@duke:/tmp# lxd sql global 'select * from images'
+----+-------------+----------+------+--------+--------------+---------------+-------------+-------------+--------+---------------+-------------+------------+------+
| id | fingerprint | filename | size | public | architecture | creation_date | expiry_date | upload_date | cached | last_use_date | auto_update | project_id | type |
+----+-------------+----------+------+--------+--------------+---------------+-------------+-------------+--------+---------------+-------------+------------+------+
+----+-------------+----------+------+--------+--------------+---------------+-------------+-------------+--------+---------------+-------------+------------+------+
root@duke:/tmp# zfs list
NAME USED AVAIL REFER MOUNTPOINT
bpool 404M 3.45G 96K /boot
bpool/BOOT 400M 3.45G 96K none
bpool/BOOT/ubuntu_npo845 399M 3.45G 164M /boot
rpool 34.1G 159G 96K /
rpool/ROOT 18.2G 159G 96K none
rpool/ROOT/ubuntu_npo845 18.2G 159G 6.35G /
rpool/ROOT/ubuntu_npo845/srv 96K 159G 96K /srv
rpool/ROOT/ubuntu_npo845/usr 3.35M 159G 96K /usr
rpool/ROOT/ubuntu_npo845/usr/local 3.26M 159G 1.59M /usr/local
rpool/ROOT/ubuntu_npo845/var 9.37G 159G 96K /var
rpool/ROOT/ubuntu_npo845/var/games 96K 159G 96K /var/games
rpool/ROOT/ubuntu_npo845/var/lib 5.24G 159G 2.82G /var/lib
rpool/ROOT/ubuntu_npo845/var/lib/AccountsService 768K 159G 96K /var/lib/AccountsService
rpool/ROOT/ubuntu_npo845/var/lib/NetworkManager 388K 159G 124K /var/lib/NetworkManager
rpool/ROOT/ubuntu_npo845/var/lib/apt 155M 159G 84.5M /var/lib/apt
rpool/ROOT/ubuntu_npo845/var/lib/dpkg 133M 159G 35.6M /var/lib/dpkg
rpool/ROOT/ubuntu_npo845/var/log 1.68G 159G 1.19G /var/log
rpool/ROOT/ubuntu_npo845/var/mail 144K 159G 120K /var/mail
rpool/ROOT/ubuntu_npo845/var/snap 2.43G 159G 18.5M /var/snap
rpool/ROOT/ubuntu_npo845/var/spool 15.8M 159G 2.95M /var/spool
rpool/ROOT/ubuntu_npo845/var/www 96K 159G 96K /var/www
rpool/USERDATA 9.48M 159G 96K /
rpool/USERDATA/root_a20v0y 9.39M 159G 6.51M /root
rpool/lxd_encrypted 12.7G 159G 192K legacy
rpool/lxd_encrypted/containers 7.64G 159G 192K legacy
rpool/lxd_encrypted/containers/testdb1 3.73G 16.3G 1.92G legacy
rpool/lxd_encrypted/containers/testdb2 3.91G 16.1G 1.87G legacy
rpool/lxd_encrypted/custom 192K 159G 192K legacy
rpool/lxd_encrypted/deleted 362M 159G 192K legacy
rpool/lxd_encrypted/deleted/containers 192K 159G 192K legacy
rpool/lxd_encrypted/deleted/custom 192K 159G 192K legacy
rpool/lxd_encrypted/deleted/images 361M 159G 192K legacy
rpool/lxd_encrypted/deleted/images/aeca0ec3057319cc132a11d5b580206de92a435cc0cd897f07152746d6fc0592 361M 159G 361M legacy
rpool/lxd_encrypted/deleted/virtual-machines 192K 159G 192K legacy
rpool/lxd_encrypted/images 361M 159G 192K legacy
rpool/lxd_encrypted/images/c1549a94399b666577d113f163194669dfb5e59291e1c35be856e73a56c8b080 361M 159G 361M legacy
rpool/lxd_encrypted/virtual-machines 4.39G 159G 192K legacy
rpool/lxd_encrypted/virtual-machines/ubuntu22install 6.34M 93.7M 6.34M legacy
rpool/lxd_encrypted/virtual-machines/ubuntu22install.block 4.38G 159G 4.38G -
rpool/lxdpool 2.90G 159G 96K none
rpool/lxdpool/containers 96K 159G 96K none
rpool/lxdpool/custom 96K 159G 96K none
rpool/lxdpool/deleted 480K 159G 96K none
rpool/lxdpool/deleted/containers 96K 159G 96K none
rpool/lxdpool/deleted/custom 96K 159G 96K none
rpool/lxdpool/deleted/images 96K 159G 96K none
rpool/lxdpool/deleted/virtual-machines 96K 159G 96K none
rpool/lxdpool/images 2.90G 159G 96K none
rpool/lxdpool/images/51fa4062f683fcd3deeac510811067c91a8fcb4c942bba9a8619bb24be1b175e 312M 159G 312M legacy
rpool/lxdpool/images/6645c75aa225491eea6c83ae7c4b94c9b936604e44828b3330f9b40b29653dc5 112K 95.3M 104K none
rpool/lxdpool/images/6645c75aa225491eea6c83ae7c4b94c9b936604e44828b3330f9b40b29653dc5.block 2.59G 159G 2.59G -
rpool/lxdpool/virtual-machines 96K 159G 96K none
root@duke:/tmp#
I’m tempted to delete the offending rows from storage_volumes but I’m afraid this will leave me with a corrupted cluster. Also there still are corresponding zfs datasets.
Any ideas?
joostn
May 11, 2022, 3:06pm
8
Oh well, I went ahead and deleted the orphaned rows from storage_volumes. It seems to have worked.