Hey,
I’m currently overhauling our backup concept and that caused a few open questions related to lxc export. We are on LXD 4.19 with lvm storage backend.
As far as I understand lxc export is creating a snapshot (not sure?), then it is copying the snapshot into a tar archive below /var/snap/lxd/common/lxd/backups/ and afterwards optionally (–compression parameter) creating a compressed tar.
Up to 4 backups per instance are kept for 24 hours. ← This could be a bug, when directing lxc export to stdout.
So my plan was to backup large containers. e.g. one with a database and 200GB+ on disk every two hours with lxc export to a borg backup repository (pipeing the lxc export stdout directly to borg stdin).
Here are my few problems:
Space: 4 times the container size for the tar archive (+optionally compressed archive)
Time: The temporary copy can take very long depending on the container size
Retention: when 4 valid backups are kept, you get an old version if you execute lxc export again.
It would be great if I could just stream the snapshot directly to borg without a temporary copy (and wasteing space). The retention behavoir should be configurable. At least I could not find anything in the docs.
As we are on lvm I already thought of just backing up a volume snapshot. But there is no non-interactive way for the restore (lxd recover is only interactive and lxd import has been removed lately). We’d like to create staging instances from the backups, what cannot be automate this way.
You can create just a snapshot on the storage pool using lxc snapshot <instance>.
The lxc export command is designed to produce a storage-pool agnostic backup of the instance.
You can also specify a target file so that the backup is immediately downloaded and then removed from the server.
This is different from lxd recover which is about recovering metadata from the storage pool if the LXD DB is damaged/removed.
The temporary storage location needed for creating the backup tarball can be changed by setting the storage.backups_volume key, e.g. lxc config set storage.backups_volume=<pool/volume> to an existing custom storage volume (can be on a different storage pool). See Server settings | LXD
The reason streaming of the temporary file isn’t supported is:
It may cause large memory spikes by the tools used to create the tarball buffering in memory if the network doesn’t keep up.
The tarball format requires having the entire file to add to the tarball ready (rather than streamed in), and in the case of optimized backups (these are blobs from supported storage pools) these are not known until the temporary file has been created (for non-optimized backups the volume is mounted and each file is added individually rather than copying the entire volume first).
Some storage pools support cheap snapshots, in those cases the export process will take a temporary snapshot and then delete it.
As you’re using LVM, you could potentially take a normal snapshot using lxc snapshot and then use the LVM volume directly (activating it if the instance isn’t running) and then either accessing the block device directly or mounting it somewhere else and streaming the filesystem contents elsewhere.
Why not omit the tar archive creation completely and stream just the files from the snapshot (+ the config metadata needed for the restore)?
Yes I completely agree. LVM snapshots would be a great way to backup.
But I could not find a way to restore this backup apart from the original container. In most cases restoring directly into a production system is not feasible. You rather create a temporary container instance and extract some files or dump some database tables.
I’m still looking for a way to create a new instance from a lvm snapshot and possibly on another node. I think there is no way to ex- and import the configuration and metadata needed for that approach. Do you have an idea how this could be done e.g. by using the rest API?
You might want to consider using a separate LXD server and then copying the instance to it from the source LXD server periodically with the --refresh flag. This will then perform an rsync.