Lxd 4.2: can no longer launch instances

1 
ubuntu@aa1-cptef101-n1:~$ lxc launch --vm --project <company>-staging2 images:debian/stretch --profile stagingtwo-2.2.25 test
Creating test
Error: Failed instance creation: Create instance from image: Failed loading image record for "3f86bf125633bfdbde9f0c6110bcaa6366e5d56d7866a764d525f39e7ef998e3": No such object

We tried deleting the image but cannot re-add it.

ubuntu@aa1-cptef101-n1:~$ lxc image ls
+-------+--------------+--------+---------------------------------------------+--------------+-----------+----------+------------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |                 DESCRIPTION                 | ARCHITECTURE |   TYPE    |   SIZE   |         UPLOAD DATE          |
+-------+--------------+--------+---------------------------------------------+--------------+-----------+----------+------------------------------+
|       | c625ffa855f2 | no     | ubuntu 20.04 LTS amd64 (release) (20200609) | x86_64       | CONTAINER | 349.44MB | Jun 24, 2020 at 3:56pm (UTC) |
+-------+--------------+--------+---------------------------------------------+--------------+-----------+----------+------------------------------+
|       | f470bf1f3996 | no     | Debian stretch amd64 (20200621_05:24)       | x86_64       | CONTAINER | 218.88MB | Jun 22, 2020 at 7:18pm (UTC) |
+-------+--------------+--------+---------------------------------------------+--------------+-----------+----------+------------------------------+
2

Can you show:

  • lxd sql global "SELECT * FROM images;"
  • lxd sql global "SELECT * FROM images_nodes;"
  • lxd sql global "SELECT * FROM storage_volumes;"

The error is coming from LXD being unable to locate the storage volume entry which is a bit odd. It could be a project-related bug, the above queries should let us sort that out.

3 
ubuntu@aa1-cptef101-n1:~$ lxd sql global "SELECT * FROM images;"
+-----+------------------------------------------------------------------+-------------------------------------------------------------------------+----------------+--------+--------------+----------------------+----------------------+--------------------------------+--------+--------------------------------+-------------+------------+------+
| id  |                           fingerprint                            |                                filename                                 |      size      | public | architecture |    creation_date     |     expiry_date      |          upload_date           | cached |         last_use_date          | auto_update | project_id | type |
+-----+------------------------------------------------------------------+-------------------------------------------------------------------------+----------------+--------+--------------+----------------------+----------------------+--------------------------------+--------+--------------------------------+-------------+------------+------+
| 188 | f470bf1f39965bc2900af9eacfccd8793c459e12515115992549e08ead95a101 | f470bf1f39965bc2900af9eacfccd8793c459e12515115992549e08ead95a101.rootfs | 2.29507748e+08 | 0      | 2            | 2020-06-21T05:30:38Z | 2020-07-21T05:30:38Z | 2020-06-22T19:18:58.613801271Z | 0      | 0001-01-01T00:00:00Z           | 0           | 1          | 0    |
| 208 | 6ec284ea013081b765b40c952f949be1d7c1233e1778983c212a2651d78dbcd3 | lxd.tar.xz                                                              | 6.8829852e+07  | 0      | 2            | 2020-06-24T00:00:00Z | 1970-01-01T00:00:00Z | 2020-06-24T09:57:08.936409144Z | 0      | 0001-01-01T00:00:00Z           | 1           | 2          | 0    |
| 214 | c625ffa855f2e84f9737259e459937ce2269902915a2fba29f7eb17250f3e032 | ubuntu-20.04-server-cloudimg-amd64-lxd.tar.xz                           | 3.6641258e+08  | 0      | 2            | 2020-06-09T00:00:00Z | 2025-04-23T00:00:00Z | 2020-06-24T15:56:45.178508764Z | 1      | 0001-01-01T00:00:00Z           | 1           | 1          | 0    |
| 219 | 3f86bf125633bfdbde9f0c6110bcaa6366e5d56d7866a764d525f39e7ef998e3 | lxd.tar.xz                                                              | 2.29442204e+08 | 0      | 2            | 2020-06-24T00:00:00Z | 1970-01-01T00:00:00Z | 2020-06-24T22:54:51.00286955Z  | 0      | 2020-06-24T23:13:26.423766186Z | 1           | 5          | 1    |
| 222 | 3f86bf125633bfdbde9f0c6110bcaa6366e5d56d7866a764d525f39e7ef998e3 | lxd.tar.xz                                                              | 2.29442204e+08 | 0      | 2            | 2020-06-24T00:00:00Z | 1970-01-01T00:00:00Z | 2020-06-24T23:07:09.205968056Z | 0      | 2020-06-24T23:13:26.423766186Z | 1           | 2          | 1    |
+-----+------------------------------------------------------------------+-------------------------------------------------------------------------+----------------+--------+--------------+----------------------+----------------------+--------------------------------+--------+--------------------------------+-------------+------------+------+
ubuntu@aa1-cptef101-n1:~$\

ubuntu@aa1-cptef101-n1:~$ lxd sql global "SELECT * FROM images_nodes;"
+-----+----------+---------+
| id  | image_id | node_id |
+-----+----------+---------+
| 228 | 188      | 2       |
| 229 | 188      | 3       |
| 230 | 188      | 1       |
| 251 | 208      | 5       |
| 257 | 214      | 5       |
| 262 | 219      | 3       |
| 265 | 222      | 3       |
+-----+----------+---------+

ubuntu@aa1-cptef101-n1:~$ lxd sql global "SELECT * FROM storage_volumes;"
[snipped 4000 lines]
| 4409 | 6ec284ea013081b765b40c952f949be1d7c1233e1778983c212a2651d78dbcd3 | 1               | 1       | 1    |             | 1          |
| 4410 | 6ec284ea013081b765b40c952f949be1d7c1233e1778983c212a2651d78dbcd3 | 1               | 2       | 1    |             | 1          |
| 4411 | 6ec284ea013081b765b40c952f949be1d7c1233e1778983c212a2651d78dbcd3 | 1               | 3       | 1    |             | 1          |
| 4412 | 6ec284ea013081b765b40c952f949be1d7c1233e1778983c212a2651d78dbcd3 | 1               | 4       | 1    |             | 1          |
| 4413 | 6ec284ea013081b765b40c952f949be1d7c1233e1778983c212a2651d78dbcd3 | 1               | 5       | 1    |             | 1          |
| 4414 | 6ec284ea013081b765b40c952f949be1d7c1233e1778983c212a2651d78dbcd3 | 1               | 6       | 1    |             | 1          |
| 4415 | 6ec284ea013081b765b40c952f949be1d7c1233e1778983c212a2651d78dbcd3 | 1               | 7       | 1    |             | 1          |
| 4416 | 6ec284ea013081b765b40c952f949be1d7c1233e1778983c212a2651d78dbcd3 | 1               | 8       | 1    |             | 1          |
| 4433 | c625ffa855f2e84f9737259e459937ce2269902915a2fba29f7eb17250f3e032 | 1               | 1       | 1    |             | 1          |
| 4434 | c625ffa855f2e84f9737259e459937ce2269902915a2fba29f7eb17250f3e032 | 1               | 2       | 1    |             | 1          |
| 4435 | c625ffa855f2e84f9737259e459937ce2269902915a2fba29f7eb17250f3e032 | 1               | 3       | 1    |             | 1          |
| 4436 | c625ffa855f2e84f9737259e459937ce2269902915a2fba29f7eb17250f3e032 | 1               | 4       | 1    |             | 1          |
| 4437 | c625ffa855f2e84f9737259e459937ce2269902915a2fba29f7eb17250f3e032 | 1               | 5       | 1    |             | 1          |
| 4438 | c625ffa855f2e84f9737259e459937ce2269902915a2fba29f7eb17250f3e032 | 1               | 6       | 1    |             | 1          |
| 4439 | c625ffa855f2e84f9737259e459937ce2269902915a2fba29f7eb17250f3e032 | 1               | 7       | 1    |             | 1          |
| 4440 | c625ffa855f2e84f9737259e459937ce2269902915a2fba29f7eb17250f3e032 | 1               | 8       | 1    |             | 1          |
+------+------------------------------------------------------------------+-----------------+---------+------+-------------+------------+
4

And what’s the right project id for your project according to lxd sql global "SELECT * FROM projects"?

5

Oh and I’ll need the lines for 3f86bf125633bfdbde9f0c6110bcaa6366e5d56d7866a764d525f39e7ef998e3 from the storage_volumes query from earlier.

6

It seems to impact both projects now.

7

That image does not show up in storage_volumes.

8

@mar can you look for any record of that image in rbd ls --pool NAME?

9

And lxc project show PROJECT would be useful too just to confirm it’s got the expected features in use.

10

Looks like I’ve reproduced the issue, trying to sort out exactly what’s going on now.

11

Looks like it does exist in ceph, at least partially. image_3f86bf125633bfdbde9f0c6110bcaa6366e5d56d7866a764d525f39e7ef998e3_ext4.block

12 
ubuntu@aa1-cptef101-n1:~$ lxc project show <company>-staging2
description: ""
config:
  features.images: "true"
  features.profiles: "true"
  features.storage.volumes: "true"
name: <company>-staging2
used_by:

and the project requiring the additional rbd images to be attached:

ubuntu@aa1-cptef101-n1:~$ lxc project show <company>-staging2-pods
description: ""
config:
  features.images: "true"
  features.profiles: "true"
  features.storage.volumes: "true"
name: <company>-staging2-pods
13

Thanks, I’ve gotten the issue reproduced even outside of ceph on a small cluster here, unfortunately that cluster is on aarch64 so I don’t have a debug build environment ready for it, building that quickly now so I can iterate against it.

14

This will take care of the general issue by having LXD handle re-tries when things get inconsistent like this.

Until this hits your version of LXD, renaming the rbd volume to have the zombie_ prefix should have LXD do the right thing.

15

Still seeing the issue:

ubuntu@aa1-cptef101-n1:~$ sudo snap refresh lxd
snap "lxd" has no updates available
ubuntu@aa1-cptef101-n1:~$ lxc launch --vm images:debian/stretch pod-test --storage ceph
Creating pod-test
Error: Failed instance creation: Create instance from image: Failed loading image record for "182978b9db6517c30a44c2220b9fed0e9b7f6a05244d8e791029c64f71a4de27": No such object
16

The fix for this issue is making its way to candidate and should hit stable on Monday.

17

Looks like the fix hit and instance creation works again now. Thanks!