Consistency of database and separated storage volumes (snapshots)

LXD
1

Bonjour,

Each night, we are lx copying a large GitLab container (server with DB), and its separated lx storage volumes (total size: 1.6 TB) from host1 to host2 (“cold” standby in case of emergency). Our hope is to be ready on host2 within minutes, should host1 fail.

We create snapshots (type container) of the server and all its attached volumes (type custom). As I understand it, a snapshot records deltas between the source dataset and the point in time the snapshot was taken (COW). Thus, to avoid any inconsistencies between the GitLab server (with its DB) and the attached volumes, we stop everything while the backup is running (‘lxc storage volume copy …’, followed by ‘lxc copy …’ to host2). However, this backup job takes too long and the downtime becomes inacceptable.

How does one handle consistency between a container snapshot and its separated storage volume snapshots? Can snapshots help here at all? Do we apply snapshots the wrong way?

Versions: Ubuntu 20.04 LTS, LXD 4.3, ZFS fs backend

Merci

2

You could do:

  • Stop container
  • Create container & storage volume snapshots
  • Start container
  • Copy the created snapshots to target server

That way your container is only offline for the time the snapshot is created which should be just a few seconds.

3

Thank you for your feedback.
So, the snapshots, while being copied over, do not try to catch up with the now again online parent datasets (i.e. delta-ing), and thus get inconsistent?

4

Correct, snapshots are always consistent.

5

We will test this out, and then report back :wink: Thank you again!