Ubuntu 16.04 server w/ 2.0.11 apt LXD. I have 12 containers using dir storage driver containing about 1.3T total.
I tried to switch LXD to snap 4.0.9 (on my way to 5.9 but lxd.migrate helpfully told me to migrate to 4.0.x first - which is good).
My current dir storage uses a separate lvm volume mounted on /var/lib/lxd. I got a warning that I would have to adjust the mount point after migration, so I thought this setup would be okay. No notice of requiring X amount of additional space before starting.
Migration started, and I noticed rsync’s running in background transferring a very large container (500G) from my existing volume to another location in the same volume. WHY? I thought it only moved the container rootfs, not copied it.
It is currently rsync’ing
/var/snap/lxd/common/lxd/containers/NAME to /var/snap/lxd/common/lxd/storage-pools/default/containers/NAME
Now I am watching the volume run out of space since it was 80% full when I started and it’s currently at 97%
So first question: once it fails, can I recover back to 2.0.11 apt (all I’ve done so far is install the snap and run lxd.migrate).
Second question: how should I have done this migration?
Thanks for listening!
$ sudo lxd.migrate │····························································
=> Connecting to source server │····························································
=> Connecting to destination server │····························································
=> Running sanity checks │····························································
│····························································
=== Source server │····························································
LXD version: 2.0.11 │····························································
LXD PID: 4479 │····························································
Resources: │····························································
Containers: 12 │····························································
Images: 6 │····························································
│····························································
=== Destination server │····························································
LXD version: 4.0.9 │····························································
LXD PID: 24987 │····························································
Resources: │····························································
Containers: 0 │····························································
Images: 0 │····························································
Networks: 0 │····························································
Storage pools: 0 │····························································
│····························································
The migration process will shut down all your containers then move your data to the destination LXD. │····························································
Once the data is moved, the destination LXD will start and apply any needed updates. │····························································
And finally your containers will be brought back to their previous state, completing the migration. │····························································
│····························································
WARNING: /var/lib/lxd is a mountpoint. You will need to update that mount location after the migration. │····························································
│····························································
Are you ready to proceed (yes/no) [default=no]? yes │····························································
=> Shutting down the source LXD │····························································
=> Stopping the source LXD units │····························································
=> Stopping the destination LXD unit │····························································
=> Unmounting source LXD paths │····························································
=> Unmounting destination LXD paths │····························································
=> Wiping destination LXD clean │····························································
=> Backing up the database │····························································
=> Moving the /var/lib/lxd mountpoint │····························································
=> Updating the storage backends │····························································
=> Starting the destination LXD │····························································
=> Waiting for LXD to come online │····························································
│····························································
At this point rsync is consuming the rest of the lvm volume mounted on
/var/snap/lxd/common/lxd
[Update after letting lxd.migrate run 24 hours]
Indeed it ran the volume out of space, but instead of dying it just hung. No errors were printed and no roll-back or cleanup was done when I eventually killed it (^C). The log files looked like it was failing on the storage init and just retrying indefinitely. It had completed migrating 1 container and made a sym link to the new snap rootfs folder, so I had to undo that move/symlink. Then I re-mounted the /var/lib/lxd, removed the partial 500G copy, removed the snap, and restored the lxd.db file.
After a reboot, I appear to be back where I started. I guess my lesson learned was to manually move /var/lib/lxd to a much larger disk first before doing another lxd.migrate. Any other tips?