This might just be me so posting for ideas before I raise it as an issue in github…
tldr
I’m getting the following error when trying to do a refresh: Error: Failed instance creation: Error transferring container data: Snapshots should not be transferred during final sync
Full Story
I have 4 servers containing a total of 12 containers.
Each night these servers snapshot each container then copy them to a backup server using:
lxc copy container backup: --refresh
This was working fine up until a few days ago when I started getting the following error intermittently…
Error: Failed instance creation: Error transferring container data: Snapshots should not be transferred during final sync
It affects all servers and all containers but is not consistent, some nights it happens with some containers other nights with others. If I rerun the refresh command in the morning most of the time it works first time, other times I get the same error and have to run it again.
I have a gut feeling it is perhaps related to the backup servers load and response times but I’m not sure.
All servers including backup server are…
BTRFS storage backend
Ubuntu 18.04.3 LTS (fully patched)
LXD 3.19 13162 stable (latest snap package)
Linux virtual1 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
There are no related errors in either dmesg, /var/snap/lxd/common/lxd/logs/lxd.log, /var/log or lxc info NAME --show-log on either the main servers or the backup server.
Anyone seen anything like this before?
I had a search for that error on both this forum and in google but didn’t find anything?
Thanks Tom.
The only other thing I thought it might be is a hardware issue on the backup server but I would expect to have seen something in a log somewhere if that was the case.
The issue will only occur in the following scenario:
Container is running
New snapshots to sync
What happens is that the running container and its new snapshot is successfully transferred to the backup server, however as the container is running it also does a ‘final sync’ of the main container volume to try and reduce any inconsistencies from the first sync. However the new snapshot does not need to have a ‘final sync’ as it is not being modified, and our new storage backend system rejects that request, however we needed to update our migration system to not request snapshots be transferred as part of the final sync step.
As the entire core LXD team is employed by Canonical we don’t currently have a way to accept donations as a project. Donating to the Ubuntu Community fund would work though. It wouldn’t be usable directly by us but those funds are used to let community members organize and attend events, so could be used by someone who wants to talk about LXD at a conference for example.