Question abot lxc export and copy

andreyche · February 13, 2023, 12:01pm

Hi, I have a question about consistency of container archive and copy of container.
Does lxc export/copy command create something like a snapshot of container?
While lxc export/copy copies, data in container are changing instantly. For example container with sql database(mysql/pg). Can I make backups with lxc export or lxc copy to secondary lxd server, will be data ok? Or I can get corrupted database?

andreyche · February 20, 2023, 12:36pm

No idea?

tomp · February 22, 2023, 8:33am

Hi, apologies for the delay in replying.

Currently today LXD will only take read-only snapshot (to provided a consistent backup) when performing an lxc export of a running instance that uses a ZFS or BTRFS storage pool.

For other storage pools the backup will be potentially inconsistent if the instance is running.

This was discussed quite a bit here:

github.com/lxc/lxd

Option to let rsync finish with errors if the operator desires

opened 12:22AM - 16 Dec 21 UTC

closed 04:14PM - 13 Jan 22 UTC

gaia

Bug Easy

* Kernel version: 5.10.0-9-amd64 Debian (source)/5.4.0-89-generic Ubuntu (target…) * LXC version: 4.0.11 | 6.1.0 (both) * LXD version: 4.21 (source)/4.0.8 (target) * Storage backend in use: dir (both`are dir drivers on underlying LVM using an entire partition) # Issue description When copying a container with `lxc copy demo-container node4: --stateless --no-profiles --instance-only -s ssd` i get `Failed instance creation: Error transferring instance data: Rsync send failed: demo-container, /var/snap/lxd/common/lxd/storage-pools/default/containers/cdemo-container/: [exit status 24 read unix @lxd/ce8bdca1-4c47-49d8-aff8-278cb5ebc919->@: use of closed network connection] (file has vanished` because as expected files are being written and deleted constantly. i do not expect a consistent copy, but the good news is that the part that is constantly changing is “self healing”, so it gets rebuilt when missing/corrupt. with this in mind, is it possible to have the instance be created, in other words, ignore any file errors? I know that standard rsync ignores these errors when copying. # Resolution Have an option to let rsync finish with errors, if the operator desires. [see this forum issue](https://discuss.linuxcontainers.org/t/lxc-copy-ignore-errors/12900)

Although I think that adding the instance “freeze” support (and optional --allow-inconsistent flag) to lxc export got lost in there.

Would you mind opening an issue at https://github.com/lxc/lxd/issues to pick this discussion back up again?

andreyche · March 24, 2023, 8:04am

Currently today LXD will only take read-only snapshot (to provided a consistent backup) when performing an lxc export of a running instance that uses a ZFS or BTRFS storage pool.

Taking snapshots is a solution, but there is a problem. When copying an instance from snapshot, flag --refresh is not supported, so I can’t make incremental copies from snapshot.

Although I think that adding the instance “freeze” support (and optional --allow-inconsistent flag) to lxc export got lost in there.

instance “freeze” isn’t supported yet?
freezing instance also has a problem, while backup is being made, the instance doesn’t work

Would be great make support of incremental backups(flag --refresh) from snapshot

tomp · April 13, 2023, 2:33pm

Well a snapshot doesn’t change, so copying from a snapshot wouldn’t need a --refresh option as it won’t change since last run.

Using lxc snapshot <instance> and then lxc copy <instance> backupserver: --refresh will (depending on storage driver) only sync the differences between the latest snapshot and the current root volume contents.

This is what BTRFS and ZFS drivers do and its what we refer to as “optimized refresh”.

hifly · April 14, 2023, 11:15am

Is there any difference in the actual commands being operated on that would prevent this also being applied to LVM? Seems like the same process to take a snapshot and then rsync it with --delete would apply?

I think this is the OPs point though. Rsync functions the same no matter the source, so could the --refresh option be supported in more cases?

tomp · May 12, 2023, 8:53am

Not all storage drivers, or even specific volume types use rsync, so any solution to implement --refresh from a snapshot would need to take a holistic view of all scenarios.

The lvm storage driver does use rsync for filesystem volumes and raw block transfer for block volumes.

Each storage driver has a property called RunningCopyFreeze which indicates whether running instances should be frozen when making (consistent, i.e when --allow-inconsistent isn’t specified) copies of them (be it for snapshots or for copying to another location).

The ZFS and BTRFS drivers have this set to false, because snapshots are deemed “cheap” enough that LXD can take a temporary one as the source of the copy to avoid needing to freeze the instance for the duration of the copy.

The LVM driver currently has RunningCopyFreeze set to true, but perhaps this could be relaxed to a conditional based on whether the pool is using LVM thin pool mode. Certainly for non-thin pools LVM snapshots incur quite an overhead so I can understand where this logic originally came from.

That being said I am not quite following the need for refreshing from a snapshot into a target instance, as this would then lose the historic snapshots on the target.

Instead you could do this:

lxc snapshot <instance> # For LVM this will briefly freeze the instance while taking the snapshot.
lxc copy <instance> backupserver: --refresh --allow-inconsistent

This will then copy the latest snapshot (which was taken consistently) and then copy the differences between that and the main volume in an inconsistent manner (to avoid freezing the instance).

You still have the latest snapshot and previous ones in a consistent manner though, and that means you can restore a specific snapshot from the backup target.