Old images not getting deleted?

pawel · May 9, 2023, 10:37am

Hi. I noticed high disk usage on one of my servers, and found out that most space is eaten by images directory. I see a lot of images there but lxc lists only two. As if it deleted old versions from database but left the files on the disk?

# ls -lt /var/snap/lxd/common/lxd/images/
total 2214908
-rw------- 1 root root  87683072 May  9 08:12 18243f857d835ef52b10b6d0646cd6affcb5f4af7f140b53bd5898908dac99cc.rootfs
-rw------- 1 root root       728 May  9 08:12 18243f857d835ef52b10b6d0646cd6affcb5f4af7f140b53bd5898908dac99cc
-rw-r--r-- 1 root root       688 May  8 08:03 0d17717351426d2b073572896cf5855e46b8f695ec284a9bd861c1904ae106bb
-rw-r--r-- 1 root root  87683072 May  8 08:03 0d17717351426d2b073572896cf5855e46b8f695ec284a9bd861c1904ae106bb.rootfs
-rw-r--r-- 1 root root       688 May  6 08:03 3fd80146c9b4b633a3399fdcb1aedc8ca98caa1e93aced236b5f78acd30cb7e7
-rw-r--r-- 1 root root  87683072 May  6 08:03 3fd80146c9b4b633a3399fdcb1aedc8ca98caa1e93aced236b5f78acd30cb7e7.rootfs
-rw-r--r-- 1 root root       688 May  4 14:03 d45fa417a7cfe2c44b9573d42acc5adefea32104c953398de6727efec8805355
-rw-r--r-- 1 root root  87678976 May  4 14:03 d45fa417a7cfe2c44b9573d42acc5adefea32104c953398de6727efec8805355.rootfs
-rw------- 1 root root 118722560 May  3 14:12 8c6dcbeb97fe51c405347e5d01f7888ab0d4f51752bee9140098ac532e6f07bc.rootfs
-rw------- 1 root root       684 May  3 14:12 8c6dcbeb97fe51c405347e5d01f7888ab0d4f51752bee9140098ac532e6f07bc
-rw-r--r-- 1 root root       688 May  2 14:03 ecc18583741445ba588fcf752df54b17efa01f8bdd0f4ae28266ae6e7d78397b
-rw-r--r-- 1 root root 118722560 May  2 14:03 ecc18583741445ba588fcf752df54b17efa01f8bdd0f4ae28266ae6e7d78397b.rootfs
-rw-r--r-- 1 root root       704 May  2 08:03 c73453a80d85d0891f3cf4c343b1a773ad5ccf3ce20d7e09725de5287e8f6945
-rw-r--r-- 1 root root  87678976 May  2 08:03 c73453a80d85d0891f3cf4c343b1a773ad5ccf3ce20d7e09725de5287e8f6945.rootfs
-rw-r--r-- 1 root root       692 Apr 30 14:03 dfa40338240decd496b0599eeb5473ba82c127d5c436c9f611308f30a10d2878
-rw-r--r-- 1 root root 118722560 Apr 30 14:03 dfa40338240decd496b0599eeb5473ba82c127d5c436c9f611308f30a10d2878.rootfs
-rw-r--r-- 1 root root       684 Apr 30 08:03 4107721295d9ca216f69a57529874613dc8a9e1851b8dc7a908ef2d51d5f934e
-rw-r--r-- 1 root root  87678976 Apr 30 08:03 4107721295d9ca216f69a57529874613dc8a9e1851b8dc7a908ef2d51d5f934e.rootfs
-rw-r--r-- 1 root root       684 Apr 27 14:03 028b5fe59e57e45766c52baa6c5d9540566a411c5b632cb25da80ffa5add84a4
-rw-r--r-- 1 root root 118665216 Apr 27 14:03 028b5fe59e57e45766c52baa6c5d9540566a411c5b632cb25da80ffa5add84a4.rootfs
-rw-r--r-- 1 root root       696 Apr 27 08:03 ce89553546c2c011788aa974a060d29c36f6439a6c70fa011a90482fcfd2ab0b
-rw-r--r-- 1 root root  87658496 Apr 27 08:03 ce89553546c2c011788aa974a060d29c36f6439a6c70fa011a90482fcfd2ab0b.rootfs
-rw-r--r-- 1 root root       684 Apr 26 02:03 c5456d253014b277702ba352ebead7e042b6de45155c61e3ace39e49e875e770
-rw-r--r-- 1 root root 118497280 Apr 26 02:03 c5456d253014b277702ba352ebead7e042b6de45155c61e3ace39e49e875e770.rootfs
-rw-r--r-- 1 root root       684 Apr 25 20:03 5909057eb156260ba0b088163be7a0917aba5a1a9a51047b664cd29506249b54
-rw-r--r-- 1 root root  87658496 Apr 25 20:03 5909057eb156260ba0b088163be7a0917aba5a1a9a51047b664cd29506249b54.rootfs
-rw-r--r-- 1 root root       684 Apr 23 20:03 d01b41c27e8666a788fe0293fbe3c873ea493c149d039fd930f84b3647bc6725
-rw-r--r-- 1 root root 118484992 Apr 23 20:03 d01b41c27e8666a788fe0293fbe3c873ea493c149d039fd930f84b3647bc6725.rootfs
-rw-r--r-- 1 root root       688 Apr 23 08:03 3bec24b944b953573603245e2b609917b1859eb7784947de3839a3fb63975142
-rw-r--r-- 1 root root  87658496 Apr 23 08:03 3bec24b944b953573603245e2b609917b1859eb7784947de3839a3fb63975142.rootfs
-rw-r--r-- 1 root root       684 Apr 21 14:03 bc361b6b684bc7a0b35e58a28d1a5197c6185f1560073d6906a00a24e151a3c7
-rw-r--r-- 1 root root 118484992 Apr 21 14:03 bc361b6b684bc7a0b35e58a28d1a5197c6185f1560073d6906a00a24e151a3c7.rootfs
-rw-r--r-- 1 root root       700 Apr 21 08:03 26a432a27c001fb51d18a5edefb35b64bbd3a84754a467d6190735ab042d88b9
-rw-r--r-- 1 root root  87658496 Apr 21 08:03 26a432a27c001fb51d18a5edefb35b64bbd3a84754a467d6190735ab042d88b9.rootfs
-rw-r--r-- 1 root root       692 Apr 19 14:03 14a21819c25939be576c3de3e86bffaf33b7ad8cab6ab632e553511d120c56b2
-rw-r--r-- 1 root root 118480896 Apr 19 14:03 14a21819c25939be576c3de3e86bffaf33b7ad8cab6ab632e553511d120c56b2.rootfs
-rw-r--r-- 1 root root       688 Apr 19 08:03 a91c794ed78fb7af1d518892a87b78dc91727c96cb0dd69fbc529c2d60172c4d
-rw-r--r-- 1 root root  87658496 Apr 19 08:03 a91c794ed78fb7af1d518892a87b78dc91727c96cb0dd69fbc529c2d60172c4d.rootfs
-rw-r--r-- 1 root root       688 Apr 17 14:03 b9299557e7419a26c01087b5c46ae6f434cb6bdd0382dbb09b8a3045d55b51b6
-rw-r--r-- 1 root root 118259712 Apr 17 14:03 b9299557e7419a26c01087b5c46ae6f434cb6bdd0382dbb09b8a3045d55b51b6.rootfs
-rw-r--r-- 1 root root       700 Apr 15 14:03 aab050a10aa01040fa4866365256a84cab0a3d58f4ae89269f893e0337b89bb3
-rw-r--r-- 1 root root 118259712 Apr 15 14:03 aab050a10aa01040fa4866365256a84cab0a3d58f4ae89269f893e0337b89bb3.rootfs
-rw-r--r-- 1 root root       684 Apr 14 09:33 14304bdc4617ea0e7e257a2762ee89ca8521b61e6d2ad192f090d9dddf7627eb
-rw-r--r-- 1 root root 118255616 Apr 14 09:33 14304bdc4617ea0e7e257a2762ee89ca8521b61e6d2ad192f090d9dddf7627eb.rootfs
# lxc image ls
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |              DESCRIPTION               | ARCHITECTURE |   TYPE    |   SIZE   |         UPLOAD DATE         |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
|       | 8c6dcbeb97fe | no     | Ubuntu focal amd64 (20230503_07:42)    | x86_64       | CONTAINER | 113.22MB | May 3, 2023 at 2:12pm (UTC) |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
|       | 18243f857d83 | no     | Debian bullseye amd64 (20230509_05:24) | x86_64       | CONTAINER | 83.62MB  | May 9, 2023 at 8:12am (UTC) |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+```

stgraber · May 9, 2023, 1:57pm

lxc project list

pawel · May 9, 2023, 2:17pm

seems that one image updated today so listing again, then follows list of projects and images per projects, seem to be the same

# ls -lt /var/snap/lxd/common/lxd/images/
total 2214932
-rw-r--r-- 1 root root       684 May  9 14:04 0234f2a1221ebf193ad98a9998d54e4231a4cf2d9d24247c2c947ef49d4ee39f
-rw-r--r-- 1 root root 118747136 May  9 14:04 0234f2a1221ebf193ad98a9998d54e4231a4cf2d9d24247c2c947ef49d4ee39f.rootfs
-rw------- 1 root root  87683072 May  9 08:12 18243f857d835ef52b10b6d0646cd6affcb5f4af7f140b53bd5898908dac99cc.rootfs
-rw------- 1 root root       728 May  9 08:12 18243f857d835ef52b10b6d0646cd6affcb5f4af7f140b53bd5898908dac99cc
-rw-r--r-- 1 root root       688 May  8 08:03 0d17717351426d2b073572896cf5855e46b8f695ec284a9bd861c1904ae106bb
-rw-r--r-- 1 root root  87683072 May  8 08:03 0d17717351426d2b073572896cf5855e46b8f695ec284a9bd861c1904ae106bb.rootfs
-rw-r--r-- 1 root root       688 May  6 08:03 3fd80146c9b4b633a3399fdcb1aedc8ca98caa1e93aced236b5f78acd30cb7e7
-rw-r--r-- 1 root root  87683072 May  6 08:03 3fd80146c9b4b633a3399fdcb1aedc8ca98caa1e93aced236b5f78acd30cb7e7.rootfs
-rw-r--r-- 1 root root       688 May  4 14:03 d45fa417a7cfe2c44b9573d42acc5adefea32104c953398de6727efec8805355
-rw-r--r-- 1 root root  87678976 May  4 14:03 d45fa417a7cfe2c44b9573d42acc5adefea32104c953398de6727efec8805355.rootfs
-rw-r--r-- 1 root root       688 May  2 14:03 ecc18583741445ba588fcf752df54b17efa01f8bdd0f4ae28266ae6e7d78397b
-rw-r--r-- 1 root root 118722560 May  2 14:03 ecc18583741445ba588fcf752df54b17efa01f8bdd0f4ae28266ae6e7d78397b.rootfs
-rw-r--r-- 1 root root       704 May  2 08:03 c73453a80d85d0891f3cf4c343b1a773ad5ccf3ce20d7e09725de5287e8f6945
-rw-r--r-- 1 root root  87678976 May  2 08:03 c73453a80d85d0891f3cf4c343b1a773ad5ccf3ce20d7e09725de5287e8f6945.rootfs
-rw-r--r-- 1 root root       692 Apr 30 14:03 dfa40338240decd496b0599eeb5473ba82c127d5c436c9f611308f30a10d2878
-rw-r--r-- 1 root root 118722560 Apr 30 14:03 dfa40338240decd496b0599eeb5473ba82c127d5c436c9f611308f30a10d2878.rootfs
-rw-r--r-- 1 root root       684 Apr 30 08:03 4107721295d9ca216f69a57529874613dc8a9e1851b8dc7a908ef2d51d5f934e
-rw-r--r-- 1 root root  87678976 Apr 30 08:03 4107721295d9ca216f69a57529874613dc8a9e1851b8dc7a908ef2d51d5f934e.rootfs
-rw-r--r-- 1 root root       684 Apr 27 14:03 028b5fe59e57e45766c52baa6c5d9540566a411c5b632cb25da80ffa5add84a4
-rw-r--r-- 1 root root 118665216 Apr 27 14:03 028b5fe59e57e45766c52baa6c5d9540566a411c5b632cb25da80ffa5add84a4.rootfs
-rw-r--r-- 1 root root       696 Apr 27 08:03 ce89553546c2c011788aa974a060d29c36f6439a6c70fa011a90482fcfd2ab0b
-rw-r--r-- 1 root root  87658496 Apr 27 08:03 ce89553546c2c011788aa974a060d29c36f6439a6c70fa011a90482fcfd2ab0b.rootfs
-rw-r--r-- 1 root root       684 Apr 26 02:03 c5456d253014b277702ba352ebead7e042b6de45155c61e3ace39e49e875e770
-rw-r--r-- 1 root root 118497280 Apr 26 02:03 c5456d253014b277702ba352ebead7e042b6de45155c61e3ace39e49e875e770.rootfs
-rw-r--r-- 1 root root       684 Apr 25 20:03 5909057eb156260ba0b088163be7a0917aba5a1a9a51047b664cd29506249b54
-rw-r--r-- 1 root root  87658496 Apr 25 20:03 5909057eb156260ba0b088163be7a0917aba5a1a9a51047b664cd29506249b54.rootfs
-rw-r--r-- 1 root root       684 Apr 23 20:03 d01b41c27e8666a788fe0293fbe3c873ea493c149d039fd930f84b3647bc6725
-rw-r--r-- 1 root root 118484992 Apr 23 20:03 d01b41c27e8666a788fe0293fbe3c873ea493c149d039fd930f84b3647bc6725.rootfs
-rw-r--r-- 1 root root       688 Apr 23 08:03 3bec24b944b953573603245e2b609917b1859eb7784947de3839a3fb63975142
-rw-r--r-- 1 root root  87658496 Apr 23 08:03 3bec24b944b953573603245e2b609917b1859eb7784947de3839a3fb63975142.rootfs
-rw-r--r-- 1 root root       684 Apr 21 14:03 bc361b6b684bc7a0b35e58a28d1a5197c6185f1560073d6906a00a24e151a3c7
-rw-r--r-- 1 root root 118484992 Apr 21 14:03 bc361b6b684bc7a0b35e58a28d1a5197c6185f1560073d6906a00a24e151a3c7.rootfs
-rw-r--r-- 1 root root       700 Apr 21 08:03 26a432a27c001fb51d18a5edefb35b64bbd3a84754a467d6190735ab042d88b9
-rw-r--r-- 1 root root  87658496 Apr 21 08:03 26a432a27c001fb51d18a5edefb35b64bbd3a84754a467d6190735ab042d88b9.rootfs
-rw-r--r-- 1 root root       692 Apr 19 14:03 14a21819c25939be576c3de3e86bffaf33b7ad8cab6ab632e553511d120c56b2
-rw-r--r-- 1 root root 118480896 Apr 19 14:03 14a21819c25939be576c3de3e86bffaf33b7ad8cab6ab632e553511d120c56b2.rootfs
-rw-r--r-- 1 root root       688 Apr 19 08:03 a91c794ed78fb7af1d518892a87b78dc91727c96cb0dd69fbc529c2d60172c4d
-rw-r--r-- 1 root root  87658496 Apr 19 08:03 a91c794ed78fb7af1d518892a87b78dc91727c96cb0dd69fbc529c2d60172c4d.rootfs
-rw-r--r-- 1 root root       688 Apr 17 14:03 b9299557e7419a26c01087b5c46ae6f434cb6bdd0382dbb09b8a3045d55b51b6
-rw-r--r-- 1 root root 118259712 Apr 17 14:03 b9299557e7419a26c01087b5c46ae6f434cb6bdd0382dbb09b8a3045d55b51b6.rootfs
-rw-r--r-- 1 root root       700 Apr 15 14:03 aab050a10aa01040fa4866365256a84cab0a3d58f4ae89269f893e0337b89bb3
-rw-r--r-- 1 root root 118259712 Apr 15 14:03 aab050a10aa01040fa4866365256a84cab0a3d58f4ae89269f893e0337b89bb3.rootfs
-rw-r--r-- 1 root root       684 Apr 14 09:33 14304bdc4617ea0e7e257a2762ee89ca8521b61e6d2ad192f090d9dddf7627eb
-rw-r--r-- 1 root root 118255616 Apr 14 09:33 14304bdc4617ea0e7e257a2762ee89ca8521b61e6d2ad192f090d9dddf7627eb.rootfs
# lxc image ls
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |              DESCRIPTION               | ARCHITECTURE |   TYPE    |   SIZE   |         UPLOAD DATE         |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
|       | 0234f2a1221e | no     | Ubuntu focal amd64 (20230509_07:42)    | x86_64       | CONTAINER | 113.25MB | May 9, 2023 at 2:04pm (UTC) |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
|       | 18243f857d83 | no     | Debian bullseye amd64 (20230509_05:24) | x86_64       | CONTAINER | 83.62MB  | May 9, 2023 at 8:12am (UTC) |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
# lxc project ls
+-------------------+--------+----------+-----------------+----------+---------------------+---------+
|       NAME        | IMAGES | PROFILES | STORAGE VOLUMES | NETWORKS |     DESCRIPTION     | USED BY |
+-------------------+--------+----------+-----------------+----------+---------------------+---------+
| 200000            | NO     | NO       | YES             | YES      |                     | 6       |
+-------------------+--------+----------+-----------------+----------+---------------------+---------+
| 200001            | NO     | NO       | YES             | YES      |                     | 6       |
+-------------------+--------+----------+-----------------+----------+---------------------+---------+
| 200002            | NO     | NO       | YES             | YES      |                     | 5       |
+-------------------+--------+----------+-----------------+----------+---------------------+---------+
| 216603            | NO     | NO       | YES             | YES      |                     | 1       |
+-------------------+--------+----------+-----------------+----------+---------------------+---------+
| default (current) | YES    | YES      | YES             | YES      | Default LXD project | 66      |
+-------------------+--------+----------+-----------------+----------+---------------------+---------+
# lxc image ls --project=200000
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |              DESCRIPTION               | ARCHITECTURE |   TYPE    |   SIZE   |         UPLOAD DATE         |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
|       | 0234f2a1221e | no     | Ubuntu focal amd64 (20230509_07:42)    | x86_64       | CONTAINER | 113.25MB | May 9, 2023 at 2:04pm (UTC) |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
|       | 18243f857d83 | no     | Debian bullseye amd64 (20230509_05:24) | x86_64       | CONTAINER | 83.62MB  | May 9, 2023 at 8:12am (UTC) |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
# lxc image ls --project=200001
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |              DESCRIPTION               | ARCHITECTURE |   TYPE    |   SIZE   |         UPLOAD DATE         |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
|       | 0234f2a1221e | no     | Ubuntu focal amd64 (20230509_07:42)    | x86_64       | CONTAINER | 113.25MB | May 9, 2023 at 2:04pm (UTC) |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
|       | 18243f857d83 | no     | Debian bullseye amd64 (20230509_05:24) | x86_64       | CONTAINER | 83.62MB  | May 9, 2023 at 8:12am (UTC) |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
# lxc image ls --project=200002
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |              DESCRIPTION               | ARCHITECTURE |   TYPE    |   SIZE   |         UPLOAD DATE         |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
|       | 0234f2a1221e | no     | Ubuntu focal amd64 (20230509_07:42)    | x86_64       | CONTAINER | 113.25MB | May 9, 2023 at 2:04pm (UTC) |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
|       | 18243f857d83 | no     | Debian bullseye amd64 (20230509_05:24) | x86_64       | CONTAINER | 83.62MB  | May 9, 2023 at 8:12am (UTC) |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
# lxc image ls --project=216603
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |              DESCRIPTION               | ARCHITECTURE |   TYPE    |   SIZE   |         UPLOAD DATE         |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
|       | 0234f2a1221e | no     | Ubuntu focal amd64 (20230509_07:42)    | x86_64       | CONTAINER | 113.25MB | May 9, 2023 at 2:04pm (UTC) |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+
|       | 18243f857d83 | no     | Debian bullseye amd64 (20230509_05:24) | x86_64       | CONTAINER | 83.62MB  | May 9, 2023 at 8:12am (UTC) |
+-------+--------------+--------+----------------------------------------+--------------+-----------+----------+-----------------------------+

stgraber · May 9, 2023, 3:29pm

lxd sql global "SELECT * FROM images"

pawel · May 10, 2023, 8:06am

# lxd sql global "SELECT * FROM images"
+----+------------------------------------------------------------------+------------+---------------+--------+--------------+----------------------+----------------------+--------------------------------+--------+----------------------+-------------+------------+------+
| id |                           fingerprint                            |  filename  |     size      | public | architecture |    creation_date     |     expiry_date      |          upload_date           | cached |    last_use_date     | auto_update | project_id | type |
+----+------------------------------------------------------------------+------------+---------------+--------+--------------+----------------------+----------------------+--------------------------------+--------+----------------------+-------------+------------+------+
| 42 | 0234f2a1221ebf193ad98a9998d54e4231a4cf2d9d24247c2c947ef49d4ee39f | lxd.tar.xz | 1.1874782e+08 | 0      | 2            | 2023-05-09T00:00:00Z | 1970-01-01T00:00:00Z | 2023-05-09T14:04:00.010003307Z | 0      | 0001-01-01T00:00:00Z | 1           | 1          | 0    |
| 43 | 448ba794b54b688ff27de03eacbe943445e4a1b4ad1764e8da980d69a1a423c6 | lxd.tar.xz | 8.768376e+07  | 0      | 2            | 2023-05-10T00:00:00Z | 1970-01-01T00:00:00Z | 2023-05-10T08:03:59.384689456Z | 0      | 0001-01-01T00:00:00Z | 1           | 1          | 0    |
+----+------------------------------------------------------------------+------------+---------------+--------+--------------+----------------------+----------------------+--------------------------------+--------+----------------------+-------------+------------+------+```

ChurchillBell · May 10, 2023, 8:23am

Check the available free space: Make sure that your device has enough free space to delete old images. If disk space is depleted, the LXC may have trouble deleting images.
Use the “lxc image delete” command: Try using the “lxc image delete” command to delete old images. Make sure that you specify the correct image name or ID when running this command.
Check your permissions: Make sure you have sufficient access rights to delete images. You may need to run the uninstall command as a user with appropriate privileges.
Check for dependencies: Check if your images have dependencies on other containers or images. If the image is used by other containers, uninstallation may be blocked. If this is the case, you will have to delete the dependent containers or images first, and then retry removing the old images.
Restart the LXC service: Sometimes restarting the LXC service can help solve image deletion problems. Try performing a restart and then retry the removal attempt.

tomp · May 12, 2023, 3:30pm

LXD should clear out any orphaned images on reload:

github.com

lxc/lxd/blob/master/lxd/images.go#L2165-L2258

      
        
            func pruneLeftoverImages(s *state.State) {
            	opRun := func(op *operations.Operation) error {
            		// Check if dealing with shared image storage.
            		var storageImages string
            		err := s.DB.Node.Transaction(context.TODO(), func(ctx context.Context, tx *db.NodeTx) error {
            			nodeConfig, err := node.ConfigLoad(ctx, tx)
            			if err != nil {
            				return err
            			}
            
            
			storageImages = nodeConfig.StorageImagesVolume()
            
            
			return nil
            		})
            		if err != nil {
            			return err
            		}
            
            
		if storageImages != "" {
            			// Parse the source.

This file has been truncated. show original

Called on startup from:

github.com

lxc/lxd/blob/399e6794b2fdbd0821f64910202478f2e25a023f/lxd/daemon.go#L1429

      
        
            		}
            	}
            
            
	// Apply all patches that need to be run after networks are initialised.
            	err = patchesApply(d, patchPostNetworks)
            	if err != nil {
            		return err
            	}
            
            
	// Cleanup leftover images.
            	pruneLeftoverImages(d.State())
            
            
	var instances []instance.Instance
            
            
	if !d.os.MockMode {
            		// Start the scheduler
            		go deviceEventListener(d.State())
            
            
		prefixPath := os.Getenv("LXD_DEVMONITOR_DIR")
            		if prefixPath == "" {
            			prefixPath = "/dev"

So does reloading LXD clear them out?

sudo systemctl reload snap.lxd.daemon

If it doesn’t then it suggests something is holding on to them.

pawel · May 16, 2023, 12:06pm

on one of the servers (i failed to mention it’s a cluster) it did the trick, on the other the files are still there

pawel · May 16, 2023, 12:06pm

lsof is not reporting any process holding on the files

pawel · May 16, 2023, 12:18pm

correction. it did work. just took a while to trigger it seems. but what now? should i reload lxd periodically to get rid of those?

tomp · May 17, 2023, 7:36am

Now that you have them all in sync please can you see if they start to accumulate again, and try and see when that occurs (anything in the logs about snap refreshing LXD for example)?