Script for backing up LXD containers

I’ve been trying to establish a container backup routine and arrived at the script below. I’m posting it here in the event others find it useful, but I’m really curious what others think about my approach, given that I am very new to LXD containers. Also, I’m not particularly skilled at writing code, but by stealing code snippets from multiple places I think I have something that works well.

Assumptions:

  • Production server (lxd_source) is an LXD server using btrfs and configured to allow remote connections
  • Backup server (lxd_target) is an LXD server using btrfs and configured to connect remotely to lxd_source
#!/bin/bash

umask 027

LXC=/usr/bin/lxc
GREP=/bin/grep
AWK=/usr/bin/awk
DATE=/bin/date
RSYNC=/usr/bin/rsync

# List of containers on lxd_source.
s_containers=`$LXC list lxd_source: -c ns --format csv | $GREP RUNNING | $AWK -F"," '{print $1}'`

# Functions
lecho () {
   logger "lxdbackup: $@"
   echo "$@"
}

#--------------------------------------
# Backup
#--------------------------------------

# For each running container on lxd_source:
#   1) Create a snapshot on lxd_source called 'current'.
#   2) If the container does not exist on lxd_target, copy the container to lxd_target, and we're done with that container.
#   3) If the container does exist on lxd_target, rsync the lxd_source snapshot to the lxd_target container.
for container in $s_containers; do
	# Note that awk on the next line is used to strip new line characters, which appear to exist and break the '-z' test.
	# If a snapshot called "current" exists for the container on lxd_source, then delete it from lxd_source.
	if [ ! -z `$LXC info lxd_source:$container | $GREP 'current (taken at' | $AWK '{print $1}'` ]; then
		lecho "Attempting to delete snapshot '$container/current' on lxd_source . . ."
		if $LXC delete lxd_source:$container/current ; then
			lecho "   Success -- deleted snapshot"
		else
			lecho "   FAILED to delete snapshot"
			lecho "   ABORTING BACKUP of container"
			continue
		fi
	fi

	# Create a snapshot of the container on lxd_source called "current".
	lecho "Attempting to create snapshot '$container/current' on lxd_source . . ."
	if $LXC snapshot lxd_source:$container current ; then
		lecho "   Success -- created snapshot"
	else
		lecho "   FAILED to create snapshot"
		lecho "   ABORTING BACKUP of container"
		continue
	fi

	# Check if container exists on lxd_target.
	if [ ! -z `$LXC list ^${container}$ -c n --format=csv` ]; then
		lecho "'$container' exists on lxd_target"
		lecho "Attempting to rsync '$container/current' on lxd_source to '$container' on lxd_target . . ."
		if $RSYNC -az -e 'ssh -i /root/.ssh/lxd_target-rsync-ssh-key' \
			root@lxd_source.example.com:/var/lib/lxd/storage-pools/default/snapshots/$container/current/ \
			/var/lib/lxd/storage-pools/default/containers/$container/.; then
			lecho "   Success -- rsync complete"
		else
			lecho "   FAILED to rsync snapshot"
			lecho "   ABORTING BACKUP of container"
			continue
		fi
	else
		lecho "'$container' does NOT exist on lxd_target"
		# Copy container/snapshot from lxd_source to lxd_target.
		lecho "Attempting to copy '$container/current' from lxd_source to lxd_target . . ."
		if $LXC copy lxd_source:$container/current $container 2>&1 > /dev/null ; then
			lecho "   Success -- copied container"
		else
			lecho "   FAILED to copy container"
			lecho "   ABORTING BACKUP of container"
			continue
		fi
		lecho "--------------------------------------------"
	fi

	# Set boot.autostart on backed up container to 'false'.
	lecho "Attempting to set boot.autostart on '$container' on lxd_target to 'false' . . ."
	if $LXC config set $container boot.autostart false 2>&1 > /dev/null ; then
		lecho "   Success -- set boot.autostart to 'false'"
	else
		lecho "   FAILED to set boot.autostart to 'false'"
	fi
	lecho "--------------------------------------------"
done

1 Like

This reminds me to my backup script that is run on the lxd host for zfs based lxc containers that allows consistent backups by rsync via container snapshots. This way I do not need to care about backup configuration on each container.

By default with the zfs storage backend the rootfs of these containers is and cannot be mounted so I run a snapshot of all containers that are not intentionally excluded from backup, mount the zfs snapshots of all those containers in a single mount directory on the lxd host filesystem, backup the mount directory by rsync to our remote backup server, then unmount again.

file lxc-backup-config

LXCBIN=/snap/bin/lxc
BACKUPMOUNTDIR=/mnt/
BACKUPTARGET=user1210@user1210.trustedspace.de
BACKUPCMD="/usr/bin/rsync -azHAX -e'ssh -p2222' --timeout=920 --inplace --numeric-ids --delete --delete-excluded -F $BACKUPMOUNTDIR $BACKUPTARGET:"

file lxc-backup.sh

#!/bin/bash

. /root/tools/lxc-backup-config

if [ -z "$BACKUPTARGET" ]; then
	echo "You need to define BACKUPTARGET in lxc-backup-config";
	exit
fi

CONTAINER_EXCLUSION_FILE=/root/tools/lxc-backup-excludes
declare -a EXCLUDED_CONTAINERS

if test -f "$CONTAINER_EXCLUSION_FILE"; then
	IFS=$'\n' read -d '' -r -a EXCLUDED_CONTAINERS < $CONTAINER_EXCLUSION_FILE
fi

echo ${EXCLUDED_CONTAINERS[@]}

isContainerExcluded () {
  local e
  for e in "${EXCLUDED_CONTAINERS[@]}"; do
  	if [ "$e" == "$1" ]; then
  		return 1;
	fi
  done
  return 0
}

CONTAINERS=`$LXCBIN ls local: --columns n --format csv`

for CONTAINER in $CONTAINERS
do
	isContainerExcluded "$CONTAINER"
	if [ $? == 0 ]; then
	    $LXCBIN delete $CONTAINER/backup 2>&1 > /dev/null
	    $LXCBIN snapshot $CONTAINER backup
	    mkdir $BACKUPMOUNTDIR$CONTAINER-backup
	    mount -t zfs tank/containers/$CONTAINER@snapshot-backup $BACKUPMOUNTDIR$CONTAINER-backup
	fi
done

eval "$BACKUPCMD"

for CONTAINER in $CONTAINERS
do
	isContainerExcluded "$CONTAINER"
	if [ $? == 0 ]; then
		umount $BACKUPMOUNTDIR$CONTAINER-backup
		rmdir $BACKUPMOUNTDIR$CONTAINER-backup
	fi
done

As you know the feature of lxc snaphot and export is usefull also if use LVM as storage backend?
I see that for production environment zfs and btrfs are recommended, but in the past I don’t have good experience to use them on Ubuntu.
Are now ready for production on bionic LTS for example?

As you know the feature of lxc snaphot and export is usefull also if use LVM as storage backend?

The problem with lxc image export is it needs additional disk space and processing time.
Assume you want to have high density on disks (I’m using up to 90% of disks for customer data) there is surely no space left for some packed image files. Furthermore those images would be deleted again after storing them on a remote machine so the generation is expendable.

I see that for production environment zfs and btrfs are recommended, but in the past I don’t have good experience to use them on Ubuntu. Are now ready for production on bionic LTS for example?

I don’t have good experiences with btrfs as well, but I’m running ZoL (ZFS on Linux) since 2014 in production and after some troubles with kernel panic on high load in the beginning it started to become absolutely stable in the last years.

1 Like

really thanks for your explanation.
Do you use Zol on Ubuntu LTS?