Backing up containers LVM storage

1

Sorry that this is a recurring subject. The available solutions aren’t good enough for me. Most solutions nowadays suggest lxc export. In my case that won’t work, the containers are way to big for that, there aren’t enough hours in a day.

I want to be able to:

  • do full and incremental backups
  • have exclude lists (like the one from rsync and tar)

My containers are in lvm storage. With LXD 3.x I was able to just use the mount point and backup from there. With the new snap LXD that is not possible anymore. The mount points aren’t visible by the native (non-snap) backup user.

What I was thinking was the following.

  • make a snapshot
  • mount the snapshot volume (read-only)
  • backup the mounted snapshot
  • unmount
  • remove the snapshot

That seems doable, however the challenge is to know what LVM volume to mount. What is the name of the snapshot volume. Is there a lxc command that could give me that name?

Right now I only can think of this method to find the device to mount. IS there an easier method?

$ sudo lxc info test04
Name: test04
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/05/19 08:10 UTC
Status: Stopped
Type: container
Profiles: pool2
Snapshots:
  backup (taken at 2020/05/19 08:14 UTC) (stateless)

Via profile and storage pool we can find the name of the LVM volume group.

lxc profile show pool2
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: pool2
    type: disk
name: pool2
used_by:
- /1.0/instances/test04

$ sudo lxc storage show pool2
config:
  lvm.thinpool_name: LXDThinPool
  lvm.vg_name: rapper-vg2
  source: rapper-vg2
  volatile.initial_source: /dev/sdc3
description: ""
name: pool2
driver: lvm
used_by:
- /1.0/containers/test04
- /1.0/containers/test04/snapshots/backup
- /1.0/images/8e28a3ed1a052dbc21aef6a22fdac1272dc14359f02f0360137280436b52b50a
- /1.0/images/c1094dfdcf5a30dec69cfc15c48eda20ea3094cd107dd74d595df53a6a13e615
- /1.0/profiles/pool2
- /1.0/profiles/pool2_20
status: Created
locations:
- none

Finally we can find out what to mount, if you know how LXD makes that volume name.

$ ls -l /dev/rapper-vg2/containers_test04-backup
lrwxrwxrwx 1 root root 49 mei 19 10:14 /dev/rapper-vg2/containers_test04-backup -> /dev/mapper/rapper--vg2-containers_test04--backup
$ ls -l /dev/mapper/rapper--vg2-containers_test04--backup
brw-rw---- 1 root disk 253, 6 mei 19 10:14 /dev/mapper/rapper--vg2-containers_test04--backup
2

A shortcut to get to the LVM volume group using lxc query

In this case I know the name of the snapshot, it’s called backup

$ sudo lxc query '/1.0/containers/test04/snapshots/backup' | jq -r '.expanded_devices.root.pool'
pool2
$ sudo lxc query '/1.0/storage-pools/pool2' | jq -r '.config."lvm.vg_name"'
rapper-vg2

Unfortunately that does always work. On another system the pool query has no config :frowning:

$ sudo lxc query /1.0/storage-pools/local
{
	"config": {},
	"description": "",
	"driver": "lvm",
	"locations": [
		"ijssel"
	],
	"name": "local",
	"status": "Created",
	"used_by": [
		"/1.0/containers/jenkins-master1",
		"/1.0/containers/jenkins-slave001",
		"/1.0/containers/jenkins-slave001/snapshots/backup",
		"/1.0/profiles/bigvol_pub",
		"/1.0/profiles/default",
		"/1.0/profiles/default_pub"
	]
}
3

Did you ever figure this out? I’m wondering the exact same thing now.

4

What are you trying to find out?

5

Sorry, I probably should have made a separate post. This post is the first result in Google for “LXD backup LVM”.

What I’m wondering is this: How do I back up files from LXD containers, without using lxc export (which uses way too much CPU + disk IO for daily backups)?

I used to use btrfs and could just back up files directly from /var/snap/lxd/common/lxd/storage-pools/default/containers/foo/rootfs/.... I used these paths with Borgbackup to do daily backups.

I migrated these containers to a system that uses LVM, and the directory under /var/snap/lxd/common/lxd/storage-pools/default/containers/foo/ just appears empty. I guess it’s namespaced or something? Honestly my experience with LVM and LXC/LXD is very limited so there might be something I’m missing.

I didn’t really find anything useful in the documentation about this. I did find this very useful blog post:

which creates a snapshot, mounts it, backs up files from the snapshot, unmounts it, then deletes it. Is that the best approach to take?

6

Based on the blog post I linked to above, I ended up writing two basic scripts - lxd-backup-mount.sh:

#!/bin/bash
set -ex

VG=chi03-vg
SOURCE_LV=$1
SNAPSHOT=backup__$SOURCE_LV
MOUNT=/mnt/lxd-backup/$SOURCE_LV

lvcreate -n $SNAPSHOT -s /dev/$VG/$SOURCE_LV
lvchange -ay -Ky $VG/$SNAPSHOT
mkdir -p $MOUNT
mount -o ro /dev/$VG/$SNAPSHOT $MOUNT

and lxd-backup-unmount.sh:

#!/bin/bash
set -ex

VG=chi03-vg
SOURCE_LV=$1

umount /mnt/lxd-backup/$SOURCE_LV
lvremove $VG/backup__$SOURCE_LV -y

which seem to work fine. Then I just need to include /mnt/lxd-backup/containers_foo/rootfs/whatever in my Borgbackup command.

1 Like
7

Yep that is a fine approach. But keep in mind we do sometimes change the LV name used for containers. Not frequently and no plans currently, but not guaranteed not to change in the future.

1 Like