LXD on LVM - best practice for disaster recovery

Although it’s officially supported, there really isn’t much documentation around for LXD on LVM, and many LXD built-in commands don’t seem to work with LVM as a storage backend. Currently, I’m trying to figure out how to best proceed to backup and, in the event of a disaster, restore my containers.

I understand there probably not the one solution for this. My primary goal is to minimize time (and complexity) to restore.

The official documentation says that in order to do a full backup, the entirety of /var/lib/lxd should be backed up, as well as the LVM volume group holding the container volumes. I’m trying to make sense of that statement.

Clearly, the content of /var/lib/lxd changes considerably depending on whether the containers are started or not. If they are started, their volumes will be mounted to /var/lib/lxd/storage-pools. If you were to back up that folder but also the whole volumes, you’d back up the same twice. So what is exactly meant here? Really back up all the data in /var/lib/lxd and just back up the meta data for the LVM volume group, or backup /var/lib/lxd without the mounted storage-pools and make a full backup of the LVM groups and volumes?

As a follow up to that question, let’s assume for a second that I cannot recreate the exact same PV/VG, I lost or discarded the meta data to do so. Instead, I switch to a new machine, set up a new PV/VG (e.g. on a bigger and better disk) but I recreate a thin pool with the same name and recreate the same volumes using backups. How would I tell LXD where to find the storage for its containers?

And any other advice you can share on this topic would be highly appreciated.

3 Likes

“Also, if you use LVM across hard drives, you may lose all your data when only one drive fails.”
(Source: server - What is LVM and what is it used for? - Ask Ubuntu, - last third).
Just a question first: Why don’t you using zfs ?
I assume you have to use LVM, right?
Which OS are you using ?
Which version of LXD ?
Do you know there is a snap LXD - THE current version - out there as well?

Before we can dive into deeper have you seen this: Backing up containers - #6 by sqllyw and this Backup the container and install it on another server - #15 by tws?

What about Ubuntu 18.04 Root on ZFS · openzfs/zfs Wiki · GitHub or
Ubuntu 16.04 Root on ZFS · openzfs/zfs Wiki · GitHub

“Also, if you use LVM across hard drives, you may lose all your data when only one drive fails.”

I know. I’m not doing this, and I have RAID anyway.

Just a question first: Why don’t you using zfs ?

I don’t see a point in using zfs on top of RAID, and LVM is a lot easier on resources.

Which OS are you using ? Which version of LXD ?

Ubuntu Bionic and LXD 3.

Before we can dive into deeper have you seen this: Backing up containers and this Backup the container and install it on another server ?

Not until now but I don’t see how any information there is more relevant than what’s in the official documentation? The info in your second link seems actually outdated, it shouldn’t be necessary to publish an image from a container in order to export it, you can run the command directly on the container.

This comment by Stephan made it clear that there simply is no easy way of doing a restore:

Restoring that is a bit harder though. You’ll need to create a /var/lib/lxd/storage-pools/POOL-NAME/containers/NAME path matching the name of the backed up container. Then if the storage pool is zfs or btrfs or lvm, you’ll need to create the applicable dataset, subvolume or lv and mount it on /var/lib/lxd/storage-pools/POOL-NAME/containers/NAME and then unpack your backup tarball onto it.

But it’s been almost 2 years since Stephan made this comment and in the meantime substantial changes were made. That’s why I thought it makes sense to ask the question again: Where do we stand today? Is restoring a lvm based LXD host (where no second LXD host for backup is being used) still a very much manual process and if yes, how does it look like today? Naturally, it would be super if some kind of automation was available but I couldn’t find anything pre-made.

with recent LXD (only available on Ubuntu with SNAP, I don’t know if it’s different on other distros), you can use lxc export (a somewhat slow process) and with the tar.gz you can do simply lxc import <path/to/file.tar.gz>, a much faster process. It works, whatever the storage or the server. I don’t see any reason it could be a problem to saving a container on a ZFS storage and restoring it on a Btrfs/lvm one.

@gpatel-fr for doing incremental backups lxc export is not sufficient. So I backup a mounted snapshot with duplicity in order to approach incremental backups for the containers

How do you backup the mounted snapshot? I think to use a similar solution to use Borg and take advantage of compression and deduplication of it.
And can I also backup the lvm thin pool and restore it after?
Now I use that system for backup lxc-containers based on LVM:
/sbin/lvcreate --snapshot --name $LVNAME-snap --size $LVSIZE $LVPATH
/sbin/borg create --compression auto,lz4 --stats --read-special $BORG_REPO::data-${START_DATE} ${LVDEV}-snap

And If I need to do a restore I can do the restore of special files using borg.