Lxd.migrate rsyncing a new copy of container during migration and running volume out of space

Ubuntu 16.04 server w/ 2.0.11 apt LXD. I have 12 containers using dir storage driver containing about 1.3T total.

I tried to switch LXD to snap 4.0.9 (on my way to 5.9 but lxd.migrate helpfully told me to migrate to 4.0.x first - which is good).

My current dir storage uses a separate lvm volume mounted on /var/lib/lxd. I got a warning that I would have to adjust the mount point after migration, so I thought this setup would be okay. No notice of requiring X amount of additional space before starting.

Migration started, and I noticed rsync’s running in background transferring a very large container (500G) from my existing volume to another location in the same volume. WHY? I thought it only moved the container rootfs, not copied it.

It is currently rsync’ing
/var/snap/lxd/common/lxd/containers/NAME to /var/snap/lxd/common/lxd/storage-pools/default/containers/NAME

Now I am watching the volume run out of space since it was 80% full when I started and it’s currently at 97%

So first question: once it fails, can I recover back to 2.0.11 apt (all I’ve done so far is install the snap and run lxd.migrate).

Second question: how should I have done this migration?

Thanks for listening!

$ sudo lxd.migrate │····························································
=> Connecting to source server │····························································
=> Connecting to destination server │····························································
=> Running sanity checks │····························································
=== Source server │····························································
LXD version: 2.0.11 │····························································
LXD PID: 4479 │····························································
Resources: │····························································
Containers: 12 │····························································
Images: 6 │····························································
=== Destination server │····························································
LXD version: 4.0.9 │····························································
LXD PID: 24987 │····························································
Resources: │····························································
Containers: 0 │····························································
Images: 0 │····························································
Networks: 0 │····························································
Storage pools: 0 │····························································
The migration process will shut down all your containers then move your data to the destination LXD. │····························································
Once the data is moved, the destination LXD will start and apply any needed updates. │····························································
And finally your containers will be brought back to their previous state, completing the migration. │····························································
WARNING: /var/lib/lxd is a mountpoint. You will need to update that mount location after the migration. │····························································
Are you ready to proceed (yes/no) [default=no]? yes │····························································
=> Shutting down the source LXD │····························································
=> Stopping the source LXD units │····························································
=> Stopping the destination LXD unit │····························································
=> Unmounting source LXD paths │····························································
=> Unmounting destination LXD paths │····························································
=> Wiping destination LXD clean │····························································
=> Backing up the database │····························································
=> Moving the /var/lib/lxd mountpoint │····························································
=> Updating the storage backends │····························································
=> Starting the destination LXD │····························································
=> Waiting for LXD to come online │····························································

At this point rsync is consuming the rest of the lvm volume mounted on

[Update after letting lxd.migrate run 24 hours]

Indeed it ran the volume out of space, but instead of dying it just hung. No errors were printed and no roll-back or cleanup was done when I eventually killed it (^C). The log files looked like it was failing on the storage init and just retrying indefinitely. It had completed migrating 1 container and made a sym link to the new snap rootfs folder, so I had to undo that move/symlink. Then I re-mounted the /var/lib/lxd, removed the partial 500G copy, removed the snap, and restored the lxd.db file.

After a reboot, I appear to be back where I started. I guess my lesson learned was to manually move /var/lib/lxd to a much larger disk first before doing another lxd.migrate. Any other tips?

The lxd.migrate tool copies the instance to the new local LXD server using the migration API.
This is why it uses rsync. It is also safer to create a copy as the old data is only removed once the migration has succeeded.

Yes, that makes sense although I don’t believe I saw any warning about the need to make a temporary copy of each container during the migration process. As a constructive suggestion from a newbie, it might be nice if lxd.migrate could calculate how much extra space it needs for migration and then check if that space is available before it starts.