Error migrating from 3.0.3 to 4.0.9

Hello

Had a stand alone server running Ubuntu 18.04 and installed lxd package version 3.0.3. Am wanting to migrate to new hardware, but was failing to copy containers to new cluster as was getting tls 1.2/1.3 issues. Advice I found online seemed to be to install the 4.0 snap and upgrade the version.

Installed 4.0 snap and ran lxd.migrate. Seemed to run ok but hung at
=> Waiting for LXD to come online

The storage pool used was a separate btrfs partition. Searching online it appeared similar to https://lxc-users.linuxcontainers.narkive.com/mJPwxsyx/error-lxd-still-not-running-after-5-minutes-failed-lxd-migrate-how-to-recover

So followed steps in that post. Adjusted fstab to point to /var/snap/lxd/common/lxd/storage-pools/default. Removed lxd debs. rebooted.

Storage pool now appears to be mounted at a sensible location but still not working. Looking at /var/snap/lxd/common/lxd/logs/lxd.log I see the following error:

t=2023-06-16T10:29:20+0000 lvl=info msg="Starting /dev/lxd handler:"
t=2023-06-16T10:29:20+0000 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2023-06-16T10:29:20+0000 lvl=info msg="REST API daemon:"
t=2023-06-16T10:29:20+0000 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
t=2023-06-16T10:29:20+0000 lvl=info msg=" - binding TCP socket" socket=[::]:8443
t=2023-06-16T10:29:20+0000 lvl=info msg="Initializing global database"
t=2023-06-16T10:29:20+0000 lvl=info msg="Connecting to global database"
t=2023-06-16T10:29:20+0000 lvl=info msg="Connected to global database"
t=2023-06-16T10:29:20+0000 lvl=info msg="Initialized global database"
t=2023-06-16T10:29:20+0000 lvl=info msg="Firewall loaded driver" driver=xtables
t=2023-06-16T10:29:20+0000 lvl=info msg="Initializing storage pools"
t=2023-06-16T10:29:20+0000 lvl=info msg="Applying patch \"storage_api_rename_container_snapshots_dir\""
t=2023-06-16T10:29:20+0000 lvl=eror msg="Failed to start the daemon" err="Failed applying patch \"storage_api_rename_container_snapshots_dir\": rename /var/snap/lxd/common/lxd/storage-pools/default/snapshots/devdocs/.migration-send /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/devdocs/.migration-send: invalid cross-device link"
t=2023-06-16T10:29:20+0000 lvl=info msg="Starting shutdown sequence" signal=interrupt

So it appears there is a migration step that still needs to run and is failing. What does storage_api_rename_container_snapshots_dir patch do? How do I fix the invalid cross-device link error?

Thanks
Will

I’d recommend deleting that .migration-send directory and try starting LXD again.

Thank you for that. Removing all the empty .migration-send folders fixed it.

Have now successfully upgraded out test server. I still need to upgrade our production one. Should I expect this to happen again? What are those .migration-send folders for?

Thanks for your help
Will