Offsite Backup for Linstor + Incus Cluster Setup

, ,
1

Hello everyone,

I have a 3-node Incus cluster (each has zfs pool) connected to a Linstor controller, where all 3 nodes are Linstor satellites with many VMs running on them.

Since the nodes share storage over Linstor if a node dies I can still access the VMs but I’m trying to plan for a total disaster scenario: what happens if all 3 nodes die at the same time?

I was planning to have offsite backups of the VMs in a bucket (Wasabi) but the Incus back up methods did not worked out for me. Snapshots are internal and export is taking the whole VM as a copy and it is not incremental. I do not want to use another Incus server for the backups. Is there any way to make me take incremental backups of the VMs in Incus?

2

Just thinking loud, do you have the chance to sync linstor snapshots somewhere?

As in btrfs / zfs support something like btrfs send | btrfs receive is there something like that with linstor?

This is what a quick research gave me:

Please also note that you have to backup the incus database as well (let’s say in case of hardware loose):

incus admin sql global .dump > incus-global-$(date).sql

Please let me know about your way to go.

3

Same boat here.

IncusOS, three compute/storage servers plus a small server for the LINSTOR controller and devops access, LINSTOR place_count=3. Looking for offsite disaster recovery to S3 Wasabi for when the whole site is gone, not Incus snapshots in the cluster.

incus export works but the tarballs are huge and slow, so that isn’t going to fly for regular backups.

We did try the LINSTOR S3 remote route. Creating the remote and taking the local snapshot is fine, but shipping to Wasabi dies on the satellite with a Java TLS error, empty trust store / cacerts on IncusOS.

Before we start hacking around that, wondering if anyone’s actually gotten linstor backup create working with Incus volumes (incremental ship to S3), and whether restore drops back into an Incus pool cleanly.

Also +1 on dumping the Incus DB, that makes sense.

4

We have all the APIs needed for proper incremental backups of containers and VMs, but that’s really on backup vendors to implement it. We have one major commercial backup vendor that’s been doing the work with an initial version coming out in the next few weeks.

And @bensmrs added support to BareOS though that predates the new APIs for incremental backups so may need some improvements to make things more efficient.

Just on the Incus side, the most reliable of using ZFS storage is to set up frequent snapshots and then script some copy --refresh to a remote disk heavy Incus server.

5

Thanks for the quick reply.

We looked at the ZFS snapshot + copy --refresh approach first, but we need encrypted disks on IncusOS and LINSTOR doesn’t work with encrypted ZFS pools, so we went LUKS (encrypt-drive) → LVM thin → LINSTOR instead.

So we’re stuck without that ZFS send path. We also tried LINSTOR’s S3 remote (linstor backup create to Wasabi): remote create and the local snapshot are fine, but shipping fails on the IncusOS satellite with a Java TLS error (empty trust store / cacerts). Before we hack around that, is the near-term recommendation a backup vendor / BareOS against the Incus APIs, or is there anything sane for LVM-thin/LINSTOR volumes (including getting LINSTOR → S3 working on IncusOS)?