I have to describe our situation first:
We run an Ubuntu system with LXD 3.0.3 that is hosting a number of different containers. The OS itself is installed on an SSD, but we also have a mechanical harddisk in the system. We have one storage pool on the SSDs for containers were performance is crucial and a different storage pool on the mechanical disk for containers where performance isn’t relevant. Both storage pools are backed by a BTRFS filesystem, native, not loop-mounted.
Every night we start a backup process that, in a loop, executes the following actions for all contaiuners on the SSD:
Create a snapshot of the running container
publish the snapshot
export the image to a directory on the HDD
delete the image from LXD
delete the snapshot
This works fine so far, but we ran into a limitation: due to the relatively small size of the SSD and some larger containers on it we ran out of space during step 2.
So the question for us is: Is it possible to publish a container’s snapshot (or a container itself) to the storage pool on the HDD instead of the one on the SSD?
I looked over the documentation, but could not find anything that would help us.
With recent LXD, we have new config options to control where the images are stored, letting you create a volume on your HDD pool and assign that as the backing for images. Publishing mostly happens within the images storage so that takes care of the problem.
On older LXD, you need to hack around things a bit, effectively setting-up a bind-mount from your HDD over /var/lib/lxd/images, making sure to transfer anything that’s in there across before doing the mount.
Thanks for your quick answer. This is a production system, due to company policy I have to stick to the LTS version, so no upgrade until the next LTS.
I thought about bind-mounting a different directory to /var/lib/lxd/images/, but then I discovered that there also exists a subvolume for every existing image under /var/lib/lxd/storage-pools/ssd/images/.
I think that contributes to our space problem, it seems to me that we have a subvolume with the image contents and the image itself on the disk. Or do I get that wrong?
You shouldn’t end up getting a subvolume for the image you just generated, those are only created when you create a container from your image. With your workflow, you only publish and then export, so that shouldn’t be the case.
The only location where your image should reside during its generation, export and until you delete it is in /var/lib/lxd/images. The images under /var/lib/lxd/storage-pools/*/images/ are those that you’ve once used to create a container or a refreshed version of such an image (ready for instant container creation).
Thanks again for your quick response. You are right, I haven’t noticed that the images under the storage-pools directory were only from containers that have been restored from an image. So until we upgrade to the next version I will go for the bind-mount solution.
Thanks again for your help, much appreciated.