Why LXD store some squashfs images outside the main storage?

All downloaded images are stored in /var/snap/lxd/common/lxd/images by default.

Then, if the storage driver supports it (and ZFS does) the first time you create an instance from an image, the image is unpacked into dedicated volume for the image on the storage driver, from there a snapshot of the cached volume is created which is your instance’s volume.

All subsequent instances created from that image on that storage driver then just snapshot the cached image volume again, without the need to do the slow unpack of the downloaded image tarball every time.

You can specify a custom volume where the download images can be stored though using storage.images_volume see https://linuxcontainers.org/lxd/docs/master/server

See Configuring backups directory for a similar example for backups.

1 Like