Why LXD store some squashfs images outside the main storage?

LXD
1

Hey,

I never noticed it because my main server has a pretty big /var, but I’m doing some tests with some machines with a very small /var and noticed that LXD is putting some squashfs images in /var/snap/lxd/common/lxd/images.

--- /var/snap/lxd/common/lxd/images -------------------------------------------------------------------------------------------------
                         /..                                                                                                         
   97.4 MiB [##########]  36e7b3c6bdeafe3260edab1ae6d4fa362c80aff36c5a3e40d4cfc526d079884f.rootfs
    4.0 KiB [          ]  36e7b3c6bdeafe3260edab1ae6d4fa362c80aff36c5a3e40d4cfc526d079884f



# zfs list
NAME                                                                            USED  AVAIL     REFER  MOUNTPOINT
local                                                                           244M  27.9G       24K  none
local/containers                                                               42.7M  27.9G       24K  none
local/containers/test02bis                                                     42.7M  27.9G      241M  /var/snap/lxd/common/lxd/storage-pools/local/containers/test02bis
local/custom                                                                     24K  27.9G       24K  none
local/deleted                                                                   120K  27.9G       24K  none
local/deleted/containers                                                         24K  27.9G       24K  none
local/deleted/custom                                                             24K  27.9G       24K  none
local/deleted/images                                                             24K  27.9G       24K  none
local/deleted/virtual-machines                                                   24K  27.9G       24K  none
local/images                                                                    201M  27.9G       24K  none
local/images/36e7b3c6bdeafe3260edab1ae6d4fa362c80aff36c5a3e40d4cfc526d079884f   201M  27.9G      201M  /var/snap/lxd/common/lxd/storage-pools/local/images/36e7b3c6bdeafe3260edab1ae6d4fa362c80aff36c5a3e40d4cfc526d079884f
local/virtual-machines                                                           24K  27.9G       24K  none

Why there is a squashfs image outside the ZFS pool?

# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
local    29G   245M  28.8G        -         -     0%     0%  1.00x    ONLINE  -
# lxc storage list
+-------+-------------+--------+---------+---------+
| NAME  | DESCRIPTION | DRIVER |  STATE  | USED BY |
+-------+-------------+--------+---------+---------+
| local |             | zfs    | CREATED | 3       |
+-------+-------------+--------+---------+---------+

Any hints? I’d like that everything to be on the ZFS local storage.

Cheers,

2

All downloaded images are stored in /var/snap/lxd/common/lxd/images by default.

Then, if the storage driver supports it (and ZFS does) the first time you create an instance from an image, the image is unpacked into dedicated volume for the image on the storage driver, from there a snapshot of the cached volume is created which is your instance’s volume.

All subsequent instances created from that image on that storage driver then just snapshot the cached image volume again, without the need to do the slow unpack of the downloaded image tarball every time.

You can specify a custom volume where the download images can be stored though using storage.images_volume see https://linuxcontainers.org/lxd/docs/master/server

See Configuring backups directory for a similar example for backups.

1 Like
3

Given storage pool is zfs, the command lxc config set storage.backups_volume bkp_pool/bkp_vol creates a zfs slice at:
…bkp_pl/custom/default_backups with mountpoint /var/snap/lxd/common/lxd/storage-pools/bkp_pl/custom/default_bkp_vol and folder /var/snap/lxd/common/lxd/backups is replaced through a symlink to /var/snap/lxd/common/lxd/storage-pools/bkp_pl/custom/default_bkp_vol.

Wouldnt it be much easier to create a zfs slice manually and just symlink the /var/snap/lxd/common/lxd/backups or /var/snap/lxd/common/lxd/images to that?
That way can assign any available host storage for backup purpose, even a cloud / nfs … storage can be mounted there which is a real external backup and not limited to lxd storage pools only.

4

You’ll find that this won’t work the way you want it as LXD runs in its own mount namespace and can’t just access all host paths.

There are ways to do this still, but it’s quite manual and not something we’ll officially support. In fact, people doing things by hand and getting into a bad spot is exactly why we implemented storage.images_volume in the first place :slight_smile:

There’s nothing wrong with you defining a new dir storage pool, point it to what you want and then use volumes from that to back your images/backups though.