Cannot find image when launching on `dir` storage backends

Peter_De_Sousa · February 9, 2023, 9:36am

Hello

When launching lxc vms or containers on my dir storage backend - I get this error message:

➜  maas-cert-lab (lxc project test ◉ : lxc launch --storage nvme-vms --vm ubuntu:focal
Creating the instance
Error: Failed instance creation: Failed creating instance from image: open /var/snap/lxd/common/lxd/images/2b6addc6b20e8f8289ae91ea392c32b5a0df3f5c2eecd508c8888f1c79ae42de: no such file or directory
➜  maas-cert-lab (lxc project berenberg-bank) ◉ : sudo ls /var/snap/lxd/common/lxd/images/

Is there any suggestion on oh how to fix this? When launching on my zfs backends, the launch works successfully.

This is the output from lxc image list:

➜  maas-cert-lab (lxc project test) ◉ : lxc image list
+-------+--------------+--------+---------------------------------------------+--------------+-----------------+----------+-----------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |                 DESCRIPTION                 | ARCHITECTURE |      TYPE       |   SIZE   |         UPLOAD DATE         |
+-------+--------------+--------+---------------------------------------------+--------------+-----------------+----------+-----------------------------+
|       | 2b6addc6b20e | no     | ubuntu 20.04 LTS amd64 (release) (20230207) | x86_64       | VIRTUAL-MACHINE | 595.44MB | Feb 9, 2023 at 9:31am (UTC) |
+-------+--------------+--------+---------------------------------------------+--------------+-----------------+----------+-----------------------------+
|       | 118ece3e492a | no     | ubuntu 20.04 LTS amd64 (release) (20230207) | x86_64       | CONTAINER       | 431.39MB | Feb 9, 2023 at 9:31am (UTC) |
+-------+--------------+--------+---------------------------------------------+--------------+-----------------+----------+-----------------------------+
|       | bd528c90d20d | no     | ubuntu 22.04 LTS amd64 (release) (20230107) | x86_64       | VIRTUAL-MACHINE | 603.19MB | Feb 9, 2023 at 9:26am (UTC) |
+-------+--------------+--------+---------------------------------------------+--------------+-----------------+----------+-----------------------------+

Any suggestions?

tomp · February 9, 2023, 11:34am

Try doing lxc image delete 2b6addc6b20e8f8289ae91ea392c32b5a0df3f5c2eecd508c8888f1c79ae42de and then retrying the launch. It should download the latest image again.

tomp · February 14, 2023, 8:09am

Can you also show lxc config show? thanks

tomp · February 14, 2023, 8:12am

Also, can you show lxc project list thanks.

tomp · February 14, 2023, 8:25am

Can you also show output of sudo lxd sql global 'select * from images'

Peter_De_Sousa · February 14, 2023, 9:16am

Thanks for the suggestions @tomp i’ll take a look

Dmitry_Lyfar · February 14, 2023, 9:33am

So, I have a similar issue that occurred for the second time, an ubuntu:20.04 server image downloaded from a LXD image server was removed but the database record of the image still persisted. Some info:

lxd 5.10-b392610 24323 latest/stable
lxc config show

config:
  core.https_address: '[::]'

The image is being used daily, hence, unlikely to be pruned or cleaned up due to expiration. @tomp would you recommend to monitor anything in particular going forward? I have another image with ubuntu:22.04 that is currently working but last time I had the issue this image has also stopped working the next day or so (they’ve been created at slightly different times).

tomp · February 14, 2023, 9:38am

Thanks.

Yes, try enabling debug logging:

sudo snap set lxd daemon.debug=true; sudo systemctl reload snap.lxd.daemon

And then checking /var/snap/lxd/common/lxd/logs/lxd.log the next time it happens for mentions of the fingerprint.

tomp · February 14, 2023, 9:53am

That will generate quite a bit of logs, depending on how busy LXD is, so just keep an eye on the size of the file too.

tomp · February 14, 2023, 10:13am

I’ve recreated the issue.

It requires using the same image in multiple projects initially, and then not using the image in one of the projects until it expires. The image record for that project is then removed and the image files, but they are still being used by the image in the other project.

tomp · February 14, 2023, 10:24am

Bug report:

github.com/lxc/lxd

Image files are incorrectly deleted when they expire in a single project (even if not expired in other projects)

opened 10:24AM - 14 Feb 23 UTC

tomponline

Bug

Reproducer: ``` lxc project create p1 -c features.images=true -c features.pr…ofiles=false lxd sql global 'select * from images' +----+-------------+----------+------+--------+--------------+---------------+-------------+-------------+--------+---------------+-------------+------------+------+ | id | fingerprint | filename | size | public | architecture | creation_date | expiry_date | upload_date | cached | last_use_date | auto_update | project_id | type | +----+-------------+----------+------+--------+--------------+---------------+-------------+-------------+--------+---------------+-------------+------------+------+ +----+-------------+----------+------+--------+--------------+---------------+-------------+-------------+--------+---------------+-------------+------------+------+ lxc init images:alpine/3.17 c1 --project default lxc init images:alpine/3.17 c2 --project p1 lxd sql global 'select * from images' +-----+------------------------------------------------------------------+------------+--------------+--------+--------------+----------------------+----------------------+--------------------------------+--------+--------------------------------+-------------+------------+------+ | id | fingerprint | filename | size | public | architecture | creation_date | expiry_date | upload_date | cached | last_use_date | auto_update | project_id | type | +-----+------------------------------------------------------------------+------------+--------------+--------+--------------+----------------------+----------------------+--------------------------------+--------+--------------------------------+-------------+------------+------+ | 293 | 6271a4dff2dc4390d66625c40b60cbcc199e38132ca21fb0929680995eebfd70 | lxd.tar.xz | 3.072876e+06 | 0 | 2 | 2023-02-13T00:00:00Z | 1970-01-01T00:00:00Z | 2023-02-14T10:19:03.043743368Z | 1 | 2023-02-14T10:19:03.077250084Z | 1 | 1 | 0 | | 295 | 6271a4dff2dc4390d66625c40b60cbcc199e38132ca21fb0929680995eebfd70 | lxd.tar.xz | 3.072876e+06 | 0 | 2 | 2023-02-13T00:00:00Z | 1970-01-01T00:00:00Z | 2023-02-14T10:20:05.334599426Z | 1 | 2023-02-14T10:20:05.369578065Z | 1 | 13 | 0 | +-----+------------------------------------------------------------------+------------+--------------+--------+--------------+----------------------+----------------------+--------------------------------+--------+--------------------------------+-------------+------------+------+ ``` Now manually update the last use date of the image record in the default project to move it far into the past so it will be considered as expired as not recently used: ``` lxd sql global 'update images set last_use_date="2023-01-01 00:00:00" where id = 293' lxd sql global 'select * from images' +-----+------------------------------------------------------------------+------------+--------------+--------+--------------+----------------------+----------------------+--------------------------------+--------+--------------------------------+-------------+------------+------+ | id | fingerprint | filename | size | public | architecture | creation_date | expiry_date | upload_date | cached | last_use_date | auto_update | project_id | type | +-----+------------------------------------------------------------------+------------+--------------+--------+--------------+----------------------+----------------------+--------------------------------+--------+--------------------------------+-------------+------------+------+ | 293 | 6271a4dff2dc4390d66625c40b60cbcc199e38132ca21fb0929680995eebfd70 | lxd.tar.xz | 3.072876e+06 | 0 | 2 | 2023-02-13T00:00:00Z | 1970-01-01T00:00:00Z | 2023-02-14T10:19:03.043743368Z | 1 | 2023-01-01T00:00:00Z | 1 | 1 | 0 | | 295 | 6271a4dff2dc4390d66625c40b60cbcc199e38132ca21fb0929680995eebfd70 | lxd.tar.xz | 3.072876e+06 | 0 | 2 | 2023-02-13T00:00:00Z | 1970-01-01T00:00:00Z | 2023-02-14T10:20:05.334599426Z | 1 | 2023-02-14T10:20:05.369578065Z | 1 | 13 | 0 | +-----+------------------------------------------------------------------+------------+--------------+--------+--------------+----------------------+----------------------+--------------------------------+--------+--------------------------------+-------------+------------+------+ ``` Restart LXD so image expired prune task runs immediately. ``` lxd sql global 'select * from images' +-----+------------------------------------------------------------------+------------+--------------+--------+--------------+----------------------+----------------------+--------------------------------+--------+--------------------------------+-------------+------------+------+ | id | fingerprint | filename | size | public | architecture | creation_date | expiry_date | upload_date | cached | last_use_date | auto_update | project_id | type | +-----+------------------------------------------------------------------+------------+--------------+--------+--------------+----------------------+----------------------+--------------------------------+--------+--------------------------------+-------------+------------+------+ | 295 | 6271a4dff2dc4390d66625c40b60cbcc199e38132ca21fb0929680995eebfd70 | lxd.tar.xz | 3.072876e+06 | 0 | 2 | 2023-02-13T00:00:00Z | 1970-01-01T00:00:00Z | 2023-02-14T10:20:05.334599426Z | 1 | 2023-02-14T10:20:05.369578065Z | 1 | 13 | 0 | +-----+------------------------------------------------------------------+------------+--------------+--------+--------------+----------------------+----------------------+--------------------------------+--------+--------------------------------+-------------+------------+------+ ``` Now the image DB record for the default project has been correctly removed, and the image DB record for the p1 project correctly remains. But, oh dear, the image files themselves on disk have been removed as part of removing the default project's DB record, rather than taking into account that the files are still referenced by a record in the p1 project. Causing: ``` lxc init images:alpine/3.17 c3 --project p1 Creating c3 Error: Failed instance creation: Failed creating instance from image: open /var/lib/lxd/images/6271a4dff2dc4390d66625c40b60cbcc199e38132ca21fb0929680995eebfd70: no such file or directory ```

tomp · February 14, 2023, 12:46pm

Fix for the issue:

Should be in LXD 5.11, released this week.

Dmitry_Lyfar · February 14, 2023, 8:23pm

Amazing, thank you @tomp

Peter_De_Sousa · February 22, 2023, 10:41am

Thanks @tomp I have 5.11 in, the images are still getting lost - how can I force a cleanup/refresh?

Peter_De_Sousa · February 22, 2023, 10:58am

Here’s my system info as you requested earlier:

lxc config show
config:
  core.bgp_address: 192.168.2.109:179
  core.bgp_asn: "65535"
  core.bgp_routerid: 192.168.2.109
  core.https_address: '[::]'
  core.trust_password: true
➜  maas-cert-lab (lxc project PROJECT ◉ : lxc project list
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
|        NAME        | IMAGES | PROFILES | STORAGE VOLUMES | STORAGE BUCKETS | NETWORKS | NETWORK ZONES |       DESCRIPTION       | USED BY |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| PROJECT            | YES    | YES      | YES             | YES             | NO       | NO            |                         | 1       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| PROJECT            | YES    | YES      | YES             | YES             | NO       | NO            |                         | 1       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| PROJECT            | YES    | YES      | YES             | YES             | NO       | NO            |                         | 7       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| bgp-lab            | YES    | YES      | YES             | YES             | NO       | NO            |                         | 1       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| charmcraft         | YES    | YES      | YES             | YES             | NO       | NO            |                         | 9       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| PROJECT            | YES    | YES      | YES             | YES             | NO       | NO            |                         | 1       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| default (current)  | YES    | YES      | YES             | YES             | YES      | YES           | Default LXD project     | 73      |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| demos              | YES    | YES      | YES             | YES             | NO       | NO            |                         | 5       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| infra              | YES    | YES      | YES             | YES             | NO       | NO            |                         | 9       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| maas-cert-lab      | YES    | YES      | YES             | YES             | NO       | NO            |                         | 3       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| openwrt-sandbox    | YES    | YES      | YES             | YES             | NO       | NO            |                         | 2       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| personal           | YES    | YES      | YES             | YES             | NO       | NO            |                         | 3       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| sandbox            | NO     | YES      | NO              | NO              | NO       | NO            | Project managed by MAAS | 1       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| sandboxing         | NO     | YES      | NO              | YES             | NO       | NO            | Project managed by MAAS | 25      |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| snapcraft          | YES    | YES      | YES             | YES             | NO       | NO            |                         | 2       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| PROJECT            | YES    | YES      | YES             | YES             | NO       | NO            |                         | 3       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| testing            | YES    | YES      | YES             | YES             | NO       | NO            |                         | 3       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| PROJECT            | YES    | YES      | YES             | YES             | NO       | NO            |                         | 3       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
| ws-sandbox         | NO     | YES      | NO              | NO              | NO       | NO            | Project managed by MAAS | 6       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-------------------------+---------+
➜  maas-cert-lab (lxc project PROJECT) ◉ : sudo lxd sql global 'select * from images'
+-----+------------------------------------------------------------------+-----------------------------------------------+----------------+--------+--------------+--------------------------------+----------------------+--------------------------------+--------+--------------------------------+-------------+------------+------+
| id  |                           fingerprint                            |                   filename                    |      size      | public | architecture |         creation_date          |     expiry_date      |          upload_date           | cached |         last_use_date          | auto_update | project_id | type |
+-----+------------------------------------------------------------------+-----------------------------------------------+----------------+--------+--------------+--------------------------------+----------------------+--------------------------------+--------+--------------------------------+-------------+------------+------+
| 4   | dcb854ea199df09c40b734d89e697d7b3344a0e10fc58c438dc3c5bf99063d52 |                                               | 5.19398639e+08 | 0      | 2            | 2022-02-10T11:07:02.923399358Z | 0001-01-01T00:00:00Z | 2022-02-10T11:07:02.926748154Z | 0      | 2022-12-12T16:43:38.036893708Z | 0           | 4          | 0    |
| 31  | ec457a69f30038b48f76e606e271760cbe4d876c642da7b415ff63b33a20e0e0 |                                               | 5.10236759e+08 | 0      | 2            | 2022-06-28T10:35:52.445387993Z | 0001-01-01T00:00:00Z | 2022-06-28T10:35:52.447004486Z | 0      | 2022-12-12T16:48:59.463362217Z | 0           | 4          | 0    |
| 159 | e90761e627debd09431b580cb87d485c4ae1f657ecceaacda17ce729156e6f7e | ubuntu-20.04-server-cloudimg-amd64-lxd.tar.xz | 4.52481856e+08 | 0      | 2            | 2023-02-09T00:00:00Z           | 2025-04-23T00:00:00Z | 2023-02-10T09:06:56.202947663Z | 0      | 2023-02-21T14:45:49.836829167Z | 1           | 1          | 0    |
| 162 | 0bff5ed9f18b9d9ff90c9f49a7071c8d87384964b1b772a86329c4a94a225837 | ubuntu-20.04-server-cloudimg-amd64-lxd.tar.xz | 6.24558912e+08 | 0      | 2            | 2023-02-09T00:00:00Z           | 2025-04-23T00:00:00Z | 2023-02-10T09:07:14.29876384Z  | 0      | 2022-12-12T13:28:03.210430195Z | 1           | 1          | 1    |
| 163 | 0bff5ed9f18b9d9ff90c9f49a7071c8d87384964b1b772a86329c4a94a225837 | ubuntu-20.04-server-cloudimg-amd64-lxd.tar.xz | 6.24558912e+08 | 0      | 2            | 2023-02-09T00:00:00Z           | 2025-04-23T00:00:00Z | 2023-02-10T09:07:14.651580508Z | 0      | 2022-12-09T16:24:28.82140549Z  | 1           | 6          | 1    |
| 164 | 0bff5ed9f18b9d9ff90c9f49a7071c8d87384964b1b772a86329c4a94a225837 | ubuntu-20.04-server-cloudimg-amd64-lxd.tar.xz | 6.24558912e+08 | 0      | 2            | 2023-02-09T00:00:00Z           | 2025-04-23T00:00:00Z | 2023-02-10T09:07:14.961868826Z | 0      | 2022-11-30T15:44:46.069389051Z | 1           | 7          | 1    |
| 168 | 095eadb433b8d65fe2f992ec774385d2961853e8d5149ae34a7f549a08e5dcb6 | ubuntu-22.04-server-cloudimg-amd64-lxd.tar.xz | 6.4893788e+08  | 0      | 2            | 2023-02-10T00:00:00Z           | 2027-04-21T00:00:00Z | 2023-02-13T09:08:50.354622543Z | 0      | 2022-11-30T15:44:38.566985832Z | 1           | 7          | 1    |
| 172 | 0bff5ed9f18b9d9ff90c9f49a7071c8d87384964b1b772a86329c4a94a225837 | ubuntu-20.04-server-cloudimg-amd64-lxd.tar.xz | 6.24558912e+08 | 0      | 2            | 2023-02-09T00:00:00Z           | 2025-04-23T00:00:00Z | 2023-02-15T11:07:44.041987277Z | 1      | 2023-02-15T11:07:53.135373781Z | 1           | 18         | 1    |
| 173 | 095eadb433b8d65fe2f992ec774385d2961853e8d5149ae34a7f549a08e5dcb6 | ubuntu-22.04-server-cloudimg-amd64-lxd.tar.xz | 6.4893788e+08  | 0      | 2            | 2023-02-10T00:00:00Z           | 2027-04-21T00:00:00Z | 2023-02-15T11:08:08.355438272Z | 1      | 2023-02-15T11:08:30.909181003Z | 1           | 18         | 1    |
| 174 | 6de3bbd44fa9fdaeef6e5a96b03bb0c09b59b6be7d594bcd3beb92c1956250b6 |                                               | 4.63298968e+08 | 0      | 2            | 2023-02-10T00:00:00Z           | 2027-04-21T00:00:00Z | 2023-02-21T14:47:11.339465722Z | 0      | 2023-02-21T14:47:11.412221895Z | 0           | 1          | 0    |
| 175 | 095eadb433b8d65fe2f992ec774385d2961853e8d5149ae34a7f549a08e5dcb6 | ubuntu-22.04-server-cloudimg-amd64-lxd.tar.xz | 6.4893788e+08  | 0      | 2            | 2023-02-10T00:00:00Z           | 2027-04-21T00:00:00Z | 2023-02-21T17:14:45.596012298Z | 1      | 2023-02-21T17:14:45.650028603Z | 1           | 1          | 1    |
| 181 | 095eadb433b8d65fe2f992ec774385d2961853e8d5149ae34a7f549a08e5dcb6 | ubuntu-22.04-server-cloudimg-amd64-lxd.tar.xz | 6.4893788e+08  | 0      | 2            | 2023-02-10T00:00:00Z           | 2027-04-21T00:00:00Z | 2023-02-22T10:55:56.389294515Z | 1      | 2023-02-22T10:55:56.434463811Z | 1           | 16         | 1    |
+-----+------------------------------------------------------------------+-----------------------------------------------+----------------+--------+--------------+--------------------------------+----------------------+--------------------------------+--------+--------------------------------+-------------+------------+------+

Peter_De_Sousa · February 22, 2023, 11:22am

here’s the error:

Creating the instance
Error: Failed instance creation: Failed creating instance from image: open /var/snap/lxd/common/lxd/images/095eadb433b8d65fe2f992ec774385d2961853e8d5149ae34a7f549a08e5dcb6: no such file or directory

tomp · February 22, 2023, 11:49am

If the image file was already deleted then LXD 5.11 won’t fix that. It just prevents them from being deleted in the future.

Suggest doing lxc image delete <fingerprint> --project <project> for the projects where the image record still exists, but the files have been removed.

Then when the next instance is created from the image it will be re-downloaded.

Peter_De_Sousa · February 22, 2023, 12:16pm

Indeed, thanks @tomp - I needed to go and do some pruning of the images, deleting the hash for each project that was using it referened in the table outputted by: sudo lxd sql global 'select * from images'. Once I removed the image from every project associated with it, lxd then went and fetched it anew from the remote.

Thanks again