I had a crash in one of my cluster members so i have reinstalled the machine, removed it from the cluster with lxd cluster --remove --force --yes and readded it with the same name
so far so good, but the images did not sync, on the reinstalled server I only have the images that were used to launch a container since reinstall
reinstalled server:
[root@lxd11 ~]# ls -al /var/snap/lxd/common/lxd/images/
total 983438
drwx------ 2 root root 4 May 25 16:45 .
drwx--x--x. 17 root root 23 May 22 18:41 ..
-rw------- 1 root root 489073017 May 25 16:45 01cded055ad49de51379ab533683abed5d94ee136ad9c0c14bf4a445a6c23a48
-rw------- 1 root root 537145683 May 22 17:37 368467b7ed1a4c3f357403f5ead7e757edb09187c3160bb27cf4cb6edc29bd72
What happens if you try and use one of the missing images as the basis for a new instance?
It should be opportunistically copied from another cluster member I believe.
I just experimented some more and realized I actually have a copy of all the images as storage volumes (I’m using zfs as backing storage if that matters) and that the images in /var/snap/led/common are only used once while importing into storage volumes and i can actually clean up /var/snap/lxd/common/lxd/images without any Ill effects. Is this correct?
If it is indeed correct, than this is actually also a case of image storage volumes not being recovered by lxd recover, should they be? It’s a bit of a moot point if they are correctly downloaded on demand, but might be easy enough to implement as a convenience.
fresh installation
root@alexsv:~# lxc image list
+-------+-------------+--------+-------------+--------------+------+------+-------------+
| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCHITECTURE | TYPE | SIZE | UPLOAD DATE |
+-------+-------------+--------+-------------+--------------+------+------+-------------+
root@alexsv:~# ls -al /var/snap/lxd/common/lxd/images/
total 34
drwx------ 2 root root 2 Mai 30 18:36 .
drwx--x--x 17 root root 24 Mai 25 11:33 ..
now lets initialize something from a public image, no need to start it
root@alexsv:~# lxc image list images: architecture=x86_64 type=container ubuntu/22.04/cloud -c lF
+-----------------------------+------------------------------------------------------------------+
| ALIAS | FINGERPRINT |
+-----------------------------+------------------------------------------------------------------+
| ubuntu/jammy/cloud (3 more) | a28b24b8bbfbcfdfa6cd129dbf573c996f6b5e0ab3f6adc289d40df09e3585d7 |
+-----------------------------+------------------------------------------------------------------+
root@alexsv:~# lxc init images:a28b24b8bbfbcfdfa6cd129dbf573c996f6b5e0ab3f6adc289d40df09e3585d7 testcontainer
Retrieving...
unpacking...
Creating testcontainer
ok, so far so good, we have an image file in /var/snap/lxd/common/lxd/images/ and a storage volume with the same thing
root@alexsv:~# ls -al /var/snap/lxd/common/lxd/images/
total 135020
drwx------ 2 root root 4 Mai 30 18:42 .
drwx--x--x 17 root root 24 Mai 25 11:33 ..
-rw-r--r-- 1 root root 704 Mai 30 18:42 a28b24b8bbfbcfdfa6cd129dbf573c996f6b5e0ab3f6adc289d40df09e3585d7
-rw-r--r-- 1 root root 138113024 Mai 30 18:42 a28b24b8bbfbcfdfa6cd129dbf573c996f6b5e0ab3f6adc289d40df09e3585d7.rootfs
root@alexsv:~# lxc storage volume list default name=a28b24b8bbfbcfdfa6cd129dbf573c996f6b5e0ab3f6adc289d40df09e3585d7
+-------+------------------------------------------------------------------+-------------+--------------+---------+
| TYPE | NAME | DESCRIPTION | CONTENT-TYPE | USED BY |
+-------+------------------------------------------------------------------+-------------+--------------+---------+
| image | a28b24b8bbfbcfdfa6cd129dbf573c996f6b5e0ab3f6adc289d40df09e3585d7 | | filesystem | 1 |
+-------+------------------------------------------------------------------+-------------+--------------+---------+
now lets remove the file and see if it lets us init again:
root@alexsv:~# rm -rf /var/snap/lxd/common/lxd/images/*
root@alexsv:~# lxc rm -f testcontainer
root@alexsv:~# lxc init local:a28b24b8bbfbcfdfa6cd129dbf573c996f6b5e0ab3f6adc289d40df09e3585d7 testcontainer
Creating testcontainer
ok, so for actual init/launch the file from /var/snap/lxd/common/lxd/images/ is not needed
lets try to copy the image somewhere else
root@alexsv:~# lxc image copy local:a28b24b8bbfbcfdfa6cd129dbf573c996f6b5e0ab3f6adc289d40df09e3585d7 lxd:
Error: Failed remote image download: open /var/snap/lxd/common/lxd/images/a28b24b8bbfbcfdfa6cd129dbf573c996f6b5e0ab3f6adc289d40df09e3585d7: no such file or directory
ok, so for copy we need the file from /var/snap/lxd/common/lxd/images/. Why not the storage volume like with init/launch? that way we can get rid of the file from /var/snap/lxd/common/lxd/images/ as soon as it is imported in the storage volume and save some storage and headaches
Right, so for instances using storage pools that support the optimized image storage feature (zfs, btrfs, lvm thin, and ceph) the first time an image is used it is unpacked into an image volume on the specific storage pool where the instance will be located. Then a writable snapshot of that image volume is taken that will be used as the root disk for the instance. Subsequent instances created using that image will just take another writable snapshot of that existing image volume. In that way it is quicker to launch future instances because the unpack operation only has to occur once.
However if you’re using a storage pool that doesn’t support optimized image storage (dir, lvm non-thin) then the image is unpacked for every instance, so it needs to be kept around. Additionally, if you create a new storage pool, or the image volume gets removed from the storage pool, then it will use the downloaded file to (re-)create image volumes by unpacking it again.
If space on the root filesystem of the host is a problem, you can instruct LXD to store the compressed image files that are downloaded on a custom volume (which isn’t the same as an image volume) on the storage pool you choose.
thanks, all is clear now, I guess I should still open a ticket so that lxc image copy triggers opportunistic download from another cluster member like lxc init does?