Source servers run lxd 2.0.11 on Ubuntu 16.04.7 LTS, deb package 2.0.11-0ubuntu1~16.04.4 using OpenZFS 0.6.5.6-0ubuntu25
Target servers run lxd 5.13 on Ubuntu 22.04.2 LTS, snap package 5.13-8e2d7eb using OpenZFS 2.1.5-1ubuntu6~22.04.1
lxc remote add says remote error: protocol version not supported
What are my options ?
Something with manual zfs send/receives ?
I don’t think the migration protocol will be compatible between those versions.
I suggest upgrading the source to the snap version (to 4.0), and then you can do lxc export <instance> <filename> transfer the file to the new hosts and do lxc import <filename>.
Hi LXD volunteers, Thomas,
Thanks for the instructions. I’ve prepared a test server with inactive copies of all the containers while still on 2.0.11 and started the upgrade there to snap install lxd --channel=3.0/stable (and wanted to hop on to 4.0 later) but ran into problems already :
20230603_004007 root@nas2:~/ /snap/bin/lxd.migrate
=> Connecting to source server
=> Connecting to destination server
=> Running sanity checks
=== Source server
LXD version: 2.0.11
LXD PID: 3624
Resources:
Containers: 14
Images: 0
=== Destination server
LXD version: 3.0.4
LXD PID: 14085
Resources:
Containers: 0
Images: 0
Networks: 0
Storage pools: 0
The migration process will shut down all your containers then move your data to the destination LXD.
Once the data is moved, the destination LXD will start and apply any needed updates.
And finally your containers will be brought back to their previous state, completing the migration.
Are you ready to proceed (yes/no) [default=no]? yes
=> Shutting down the source LXD
=> Stopping the source LXD units
=> Stopping the destination LXD unit
=> Unmounting source LXD paths
=> Unmounting destination LXD paths
=> Wiping destination LXD clean
=> Backing up the database
=> Moving the data
=> Updating the storage backends
=> Starting the destination LXD
=> Waiting for LXD to come online
And it is stuck there.
The lxd.migrate process is trying to connect to /var/snap/lxd/common/lxd/unix.socket which never succeeds.
20230603_004533 hans@nas2:~/ sudo strace -p 13923
strace: Process 13923 attached
restart_syscall(<... resuming interrupted futex ...>) = 0
futex(0x10c3c40, FUTEX_WAIT_PRIVATE, 0, {40, 315543387}) = 0
futex(0x10bffb0, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x10bfeb0, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x10c3c40, FUTEX_WAIT_PRIVATE, 0, {0, 498589936}) = -1 ETIMEDOUT (Connection timed out)
futex(0x10bffb0, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x10bfeb0, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000022bc8, FUTEX_WAKE_PRIVATE, 1) = 1
socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
connect(3, {sa_family=AF_LOCAL, sun_path="/var/snap/lxd/common/lxd/unix.socket"}, 39) = -1 ECONNREFUSED (Connection refused)
close(3) = 0
futex(0xc0001812c8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x10c0828, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x10c0828, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x10c0828, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x10c0828, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x10c0828, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x10c0828, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x10c0828, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x10c0828, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x10c0828, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x10c0828, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
connect(3, {sa_family=AF_LOCAL, sun_path="/var/snap/lxd/common/lxd/unix.socket"}, 39) = -1 ECONNREFUSED (Connection refused)
close(3) = 0
futex(0xc000022bc8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x10c0828, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x10c0828, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x10c0828, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
Syslog shows this :
20230603_010022 root@nas2:~/ grep lxd /var/log/syslog
Jun 3 00:38:08 nas2 snapd[12847]: api.go:1007: Installing snap "lxd" revision unset
Jun 3 00:38:42 nas2 systemd[1]: [/etc/systemd/system/snap-lxd-11348.mount:12] Unknown lvalue 'LazyUnmount' in section 'Mount'
Jun 3 00:38:42 nas2 systemd[1]: Mounting Mount unit for lxd, revision 11348...
Jun 3 00:38:42 nas2 systemd[1]: Mounted Mount unit for lxd, revision 11348.
Jun 3 00:38:43 nas2 kernel: [53455135.963187] audit: type=1400 audit(1685745523.524:47913): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap-update-ns.lxd" pid=13380 comm="apparmor_parser"
Jun 3 00:38:43 nas2 kernel: [53455136.282190] audit: type=1400 audit(1685745523.844:47914): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.activate" pid=13381 comm="apparmor_parser"
Jun 3 00:38:44 nas2 kernel: [53455136.603321] audit: type=1400 audit(1685745524.168:47915): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.benchmark" pid=13384 comm="apparmor_parser"
Jun 3 00:38:44 nas2 kernel: [53455136.912187] audit: type=1400 audit(1685745524.476:47916): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.buginfo" pid=13387 comm="apparmor_parser"
Jun 3 00:38:44 nas2 kernel: [53455137.221170] audit: type=1400 audit(1685745524.784:47917): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.check-kernel" pid=13390 comm="apparmor_parser"
Jun 3 00:38:45 nas2 kernel: [53455137.534924] audit: type=1400 audit(1685745525.096:47918): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.daemon" pid=13393 comm="apparmor_parser"
Jun 3 00:38:45 nas2 kernel: [53455137.851144] audit: type=1400 audit(1685745525.412:47919): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.hook.configure" pid=13396 comm="apparmor_parser"
Jun 3 00:38:45 nas2 kernel: [53455138.158610] audit: type=1400 audit(1685745525.720:47920): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.hook.install" pid=13399 comm="apparmor_parser"
Jun 3 00:38:46 nas2 kernel: [53455138.464512] audit: type=1400 audit(1685745526.028:47921): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.lxc" pid=13402 comm="apparmor_parser"
Jun 3 00:38:46 nas2 kernel: [53455138.773911] audit: type=1400 audit(1685745526.336:47922): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.lxd" pid=13405 comm="apparmor_parser"
Jun 3 00:38:46 nas2 systemd[1]: [/etc/systemd/system/snap-lxd-11348.mount:12] Unknown lvalue 'LazyUnmount' in section 'Mount'
Jun 3 00:38:48 nas2 kernel: [53455141.402262] audit: type=1400 audit(1685745528.964:47929): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.buginfo" pid=13486 comm="apparmor_parser"
Jun 3 00:38:49 nas2 kernel: [53455141.761550] audit: type=1400 audit(1685745529.324:47930): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.check-kernel" pid=13487 comm="apparmor_parser"
Jun 3 00:38:49 nas2 kernel: [53455142.201109] audit: type=1400 audit(1685745529.764:47931): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.daemon" pid=13492 comm="apparmor_parser"
Jun 3 00:38:50 nas2 kernel: [53455142.590227] audit: type=1400 audit(1685745530.152:47932): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.hook.configure" pid=13495 comm="apparmor_parser"
Jun 3 00:38:50 nas2 kernel: [53455142.902156] audit: type=1400 audit(1685745530.464:47933): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.hook.install" pid=13498 comm="apparmor_parser"
Jun 3 00:38:50 nas2 kernel: [53455143.255857] audit: type=1400 audit(1685745530.820:47934): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.lxc" pid=13501 comm="apparmor_parser"
Jun 3 00:38:51 nas2 kernel: [53455143.602494] audit: type=1400 audit(1685745531.164:47935): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.lxd" pid=13504 comm="apparmor_parser"
Jun 3 00:38:51 nas2 kernel: [53455143.951213] audit: type=1400 audit(1685745531.512:47936): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.migrate" pid=13507 comm="apparmor_parser"
Jun 3 00:38:52 nas2 systemd[1]: Started snap.lxd.hook.install.1d4cb408-1ba6-4897-a4d5-cdd1a2fd154e.scope.
Jun 3 00:38:53 nas2 systemd[1]: [/etc/systemd/system/snap-lxd-11348.mount:12] Unknown lvalue 'LazyUnmount' in section 'Mount'
Jun 3 00:38:53 nas2 systemd[1]: Listening on Socket unix for snap application lxd.daemon.
Jun 3 00:38:53 nas2 systemd[1]: Starting Service for snap application lxd.activate...
Jun 3 00:38:53 nas2 lxd.activate[13562]: => Starting LXD activation
Jun 3 00:38:53 nas2 lxd.activate[13562]: ==> Creating missing snap configuration
Jun 3 00:38:54 nas2 lxd.activate[13562]: ==> Loading snap configuration
Jun 3 00:38:54 nas2 lxd.activate[13562]: ==> Checking for socket activation support
Jun 3 00:38:54 nas2 lxd.activate[13562]: ==> Setting LXD socket ownership
Jun 3 00:38:54 nas2 lxd.activate[13562]: ==> LXD never started on this system, no need to start it now
Jun 3 00:38:54 nas2 systemd[1]: Started Service for snap application lxd.activate.
Jun 3 00:38:54 nas2 systemd[1]: Started snap.lxd.hook.configure.d7ca0dcc-f44a-4ef3-ab37-0a7b2d318048.scope.
Jun 3 00:39:26 nas2 systemd[1]: Started snap.lxd.lxc.eef90eb6-f60e-415f-b729-13f8a0df5f95.scope.
Jun 3 00:40:19 nas2 systemd[1]: Started snap.lxd.migrate.7b06022b-a83f-44da-8c32-11f10a3edf8e.scope.
Jun 3 00:40:20 nas2 systemd[1]: Started Service for snap application lxd.daemon.
Jun 3 00:40:20 nas2 lxd.daemon[13958]: => Preparing the system
Jun 3 00:40:20 nas2 lxd.daemon[13958]: ==> Loading snap configuration
Jun 3 00:40:20 nas2 lxd.daemon[13958]: ==> Setting up mntns symlink (mnt:[4026532238])
Jun 3 00:40:20 nas2 lxd.daemon[13958]: ==> Setting up persistent shmounts path
Jun 3 00:40:20 nas2 lxd.daemon[13958]: ====> Making LXD shmounts use the persistent path
Jun 3 00:40:20 nas2 lxd.daemon[13958]: ====> Making LXCFS use the persistent path
Jun 3 00:40:20 nas2 lxd.daemon[13958]: ==> Setting up kmod wrapper
Jun 3 00:40:20 nas2 lxd.daemon[13958]: ==> Preparing /boot
Jun 3 00:40:20 nas2 lxd.daemon[13958]: ==> Preparing a clean copy of /run
Jun 3 00:40:20 nas2 lxd.daemon[13958]: ==> Preparing a clean copy of /etc
Jun 3 00:40:21 nas2 lxd.daemon[13958]: ==> Setting up bash completion
Jun 3 00:40:21 nas2 lxd.daemon[13958]: ==> Setting up ceph configuration
Jun 3 00:40:21 nas2 lxd.daemon[13958]: ==> Setting up LVM configuration
Jun 3 00:40:21 nas2 lxd.daemon[13958]: ==> Rotating logs
Jun 3 00:40:21 nas2 lxd.daemon[13958]: ==> Setting up ZFS (0.6)
Jun 3 00:40:21 nas2 lxd.daemon[13958]: ==> Escaping the systemd cgroups
Jun 3 00:40:21 nas2 lxd.daemon[13958]: ==> Escaping the systemd process resource limits
Jun 3 00:40:21 nas2 lxd.daemon[13958]: => Starting LXCFS
Jun 3 00:40:21 nas2 lxd.daemon[13958]: mount namespace: 5
Jun 3 00:40:21 nas2 lxd.daemon[13958]: hierarchies:
Jun 3 00:40:21 nas2 lxd.daemon[13958]: 0: fd: 6: blkio
Jun 3 00:40:21 nas2 lxd.daemon[13958]: 1: fd: 7: hugetlb
Jun 3 00:40:21 nas2 lxd.daemon[13958]: 2: fd: 8: pids
Jun 3 00:40:21 nas2 lxd.daemon[13958]: 3: fd: 9: devices
Jun 3 00:40:21 nas2 lxd.daemon[13958]: 4: fd: 10: net_cls,net_prio
Jun 3 00:40:21 nas2 lxd.daemon[13958]: 5: fd: 11: memory
Jun 3 00:40:21 nas2 lxd.daemon[13958]: 6: fd: 12: cpuset
Jun 3 00:40:21 nas2 lxd.daemon[13958]: 7: fd: 13: perf_event
Jun 3 00:40:21 nas2 lxd.daemon[13958]: 8: fd: 14: cpu,cpuacct
Jun 3 00:40:21 nas2 lxd.daemon[13958]: 9: fd: 15: freezer
Jun 3 00:40:21 nas2 lxd.daemon[13958]: 10: fd: 16: name=systemd
Jun 3 00:40:21 nas2 lxd.daemon[13958]: => Starting LXD
Jun 3 00:40:23 nas2 lxd.daemon[13958]: t=2023-06-03T00:40:23+0200 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored."
Jun 3 00:40:25 nas2 lxd.daemon[13958]: => LXD is ready
Jun 3 00:41:28 nas2 systemd[1]: Closed Socket unix for snap application lxd.daemon.
Jun 3 00:41:29 nas2 systemd[1]: Stopping Service for snap application lxd.daemon...
Jun 3 00:41:29 nas2 lxd.daemon[14227]: => Stop reason is: host shutdown
Jun 3 00:41:29 nas2 lxd.daemon[14227]: => Stopping LXD (with container shutdown)
Jun 3 00:41:29 nas2 lxd.daemon[13958]: => LXD exited cleanly
Jun 3 00:41:29 nas2 lxd.daemon[14227]: => Stopping LXCFS
Jun 3 00:41:30 nas2 systemd[1]: Stopped Service for snap application lxd.daemon.
Jun 3 00:41:35 nas2 systemd[1]: Started Service for snap application lxd.daemon.
Jun 3 00:41:35 nas2 lxd.daemon[14306]: => Preparing the system
Jun 3 00:41:35 nas2 lxd.daemon[14306]: ==> Loading snap configuration
Jun 3 00:41:35 nas2 lxd.daemon[14306]: ==> Setting up mntns symlink (mnt:[4026532238])
Jun 3 00:41:35 nas2 lxd.daemon[14306]: ==> Setting up kmod wrapper
Jun 3 00:41:35 nas2 lxd.daemon[14306]: ==> Preparing /boot
Jun 3 00:41:35 nas2 lxd.daemon[14306]: ==> Preparing a clean copy of /run
Jun 3 00:41:35 nas2 lxd.daemon[14306]: ==> Preparing a clean copy of /etc
Jun 3 00:41:35 nas2 lxd.daemon[14306]: ==> Setting up ceph configuration
Jun 3 00:41:35 nas2 lxd.daemon[14306]: ==> Setting up LVM configuration
Jun 3 00:41:35 nas2 lxd.daemon[14306]: ==> Rotating logs
Jun 3 00:41:35 nas2 lxd.daemon[14306]: ==> Setting up ZFS (0.6)
Jun 3 00:41:35 nas2 lxd.daemon[14306]: ==> Escaping the systemd cgroups
Jun 3 00:41:35 nas2 lxd.daemon[14306]: ==> Escaping the systemd process resource limits
Jun 3 00:41:35 nas2 lxd.daemon[14306]: => Starting LXCFS
Jun 3 00:41:35 nas2 lxd.daemon[14306]: mount namespace: 5
Jun 3 00:41:35 nas2 lxd.daemon[14306]: hierarchies:
Jun 3 00:41:35 nas2 lxd.daemon[14306]: 0: fd: 6: blkio
Jun 3 00:41:35 nas2 lxd.daemon[14306]: 1: fd: 7: hugetlb
Jun 3 00:41:35 nas2 lxd.daemon[14306]: 2: fd: 8: pids
Jun 3 00:41:35 nas2 lxd.daemon[14306]: 3: fd: 9: devices
Jun 3 00:41:35 nas2 lxd.daemon[14306]: 4: fd: 10: net_cls,net_prio
Jun 3 00:41:35 nas2 lxd.daemon[14306]: 5: fd: 11: memory
Jun 3 00:41:35 nas2 lxd.daemon[14306]: 6: fd: 12: cpuset
Jun 3 00:41:35 nas2 lxd.daemon[14306]: 7: fd: 13: perf_event
Jun 3 00:41:35 nas2 lxd.daemon[14306]: 8: fd: 14: cpu,cpuacct
Jun 3 00:41:35 nas2 lxd.daemon[14306]: 9: fd: 15: freezer
Jun 3 00:41:35 nas2 lxd.daemon[14306]: 10: fd: 16: name=systemd
Jun 3 00:41:35 nas2 lxd.daemon[14306]: => Starting LXD
Jun 3 00:41:35 nas2 lxd.daemon[14306]: t=2023-06-03T00:41:35+0200 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored."
Jun 3 00:41:35 nas2 lxd.daemon[14306]: t=2023-06-03T00:41:35+0200 lvl=eror msg="Failed to start the daemon: Both legacy and new local database files exists"
Jun 3 00:41:35 nas2 lxd.daemon[14306]: Error: Both legacy and new local database files exists
Jun 3 00:41:36 nas2 lxd.daemon[14306]: => LXD failed to start
Jun 3 00:41:36 nas2 systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Jun 3 00:41:36 nas2 lxd.daemon[14431]: => Stop reason is: crashed
Jun 3 00:41:36 nas2 systemd[1]: snap.lxd.daemon.service: Unit entered failed state.
Jun 3 00:41:36 nas2 systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Jun 3 00:41:37 nas2 systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Jun 3 00:41:37 nas2 systemd[1]: Stopped Service for snap application lxd.daemon.
Jun 3 00:41:37 nas2 systemd[1]: Started Service for snap application lxd.daemon.
Jun 3 00:41:37 nas2 lxd.daemon[14465]: => Preparing the system
Jun 3 00:41:37 nas2 lxd.daemon[14465]: ==> Loading snap configuration
Jun 3 00:41:37 nas2 lxd.daemon[14465]: ==> Setting up mntns symlink (mnt:[4026532238])
Jun 3 00:41:37 nas2 lxd.daemon[14465]: ==> Setting up kmod wrapper
Jun 3 00:41:37 nas2 lxd.daemon[14465]: ==> Preparing /boot
Jun 3 00:41:37 nas2 lxd.daemon[14465]: ==> Preparing a clean copy of /run
Jun 3 00:41:37 nas2 lxd.daemon[14465]: ==> Preparing a clean copy of /etc
Jun 3 00:41:37 nas2 lxd.daemon[14465]: ==> Setting up ceph configuration
Jun 3 00:41:37 nas2 lxd.daemon[14465]: ==> Setting up LVM configuration
Jun 3 00:41:37 nas2 lxd.daemon[14465]: ==> Rotating logs
Jun 3 00:41:37 nas2 lxd.daemon[14465]: ==> Setting up ZFS (0.6)
Jun 3 00:41:37 nas2 lxd.daemon[14465]: ==> Escaping the systemd cgroups
Jun 3 00:41:37 nas2 lxd.daemon[14465]: ==> Escaping the systemd process resource limits
Jun 3 00:41:37 nas2 lxd.daemon[14465]: => Re-using existing LXCFS
Jun 3 00:41:37 nas2 lxd.daemon[14465]: => Starting LXD
Jun 3 00:41:37 nas2 lxd.daemon[14465]: t=2023-06-03T00:41:37+0200 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored."
Jun 3 00:41:37 nas2 lxd.daemon[14465]: t=2023-06-03T00:41:37+0200 lvl=eror msg="Failed to start the daemon: Both legacy and new local database files exists"
Jun 3 00:41:37 nas2 lxd.daemon[14465]: Error: Both legacy and new local database files exists
Jun 3 00:41:38 nas2 lxd.daemon[14465]: => LXD failed to start
Jun 3 00:41:38 nas2 systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Jun 3 00:41:38 nas2 lxd.daemon[14584]: => Stop reason is: crashed
Jun 3 00:41:38 nas2 systemd[1]: snap.lxd.daemon.service: Unit entered failed state.
Jun 3 00:41:38 nas2 systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Jun 3 00:41:38 nas2 systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Jun 3 00:41:38 nas2 systemd[1]: Stopped Service for snap application lxd.daemon.
Jun 3 00:41:38 nas2 systemd[1]: Started Service for snap application lxd.daemon.
Jun 3 00:41:39 nas2 lxd.daemon[14615]: => Preparing the system
Jun 3 00:41:39 nas2 lxd.daemon[14615]: ==> Loading snap configuration
Jun 3 00:41:39 nas2 lxd.daemon[14615]: ==> Setting up mntns symlink (mnt:[4026532238])
Jun 3 00:41:39 nas2 lxd.daemon[14615]: ==> Setting up kmod wrapper
Jun 3 00:41:39 nas2 lxd.daemon[14615]: ==> Preparing /boot
Jun 3 00:41:39 nas2 lxd.daemon[14615]: ==> Preparing a clean copy of /run
Jun 3 00:41:39 nas2 lxd.daemon[14615]: ==> Preparing a clean copy of /etc
Jun 3 00:41:39 nas2 lxd.daemon[14615]: ==> Setting up ceph configuration
Jun 3 00:41:39 nas2 lxd.daemon[14615]: ==> Setting up LVM configuration
Jun 3 00:41:39 nas2 lxd.daemon[14615]: ==> Rotating logs
Jun 3 00:41:39 nas2 lxd.daemon[14615]: ==> Setting up ZFS (0.6)
Jun 3 00:41:39 nas2 lxd.daemon[14615]: ==> Escaping the systemd cgroups
Jun 3 00:41:39 nas2 lxd.daemon[14615]: ==> Escaping the systemd process resource limits
Jun 3 00:41:39 nas2 lxd.daemon[14306]: mount namespace: 7
Jun 3 00:41:39 nas2 lxd.daemon[14306]: hierarchies:
Jun 3 00:41:39 nas2 lxd.daemon[14306]: 0: fd: 8: blkio
Jun 3 00:41:39 nas2 lxd.daemon[14306]: 1: fd: 9: hugetlb
Jun 3 00:41:39 nas2 lxd.daemon[14306]: 2: fd: 10: pids
Jun 3 00:41:39 nas2 lxd.daemon[14306]: 3: fd: 11: devices
Jun 3 00:41:39 nas2 lxd.daemon[14306]: 4: fd: 12: net_cls,net_prio
Jun 3 00:41:39 nas2 lxd.daemon[14306]: 5: fd: 13: memory
Jun 3 00:41:39 nas2 lxd.daemon[14306]: 6: fd: 14: cpuset
Jun 3 00:41:39 nas2 lxd.daemon[14306]: 7: fd: 15: perf_event
Jun 3 00:41:39 nas2 lxd.daemon[14306]: 8: fd: 16: cpu,cpuacct
Jun 3 00:41:39 nas2 lxd.daemon[14306]: 9: fd: 17: freezer
Jun 3 00:41:39 nas2 lxd.daemon[14306]: 10: fd: 18: name=systemd
Jun 3 00:41:39 nas2 lxd.daemon[14306]: lxcfs.c: 105: do_reload: lxcfs: reloaded
Jun 3 00:41:39 nas2 lxd.daemon[14615]: => Re-using existing LXCFS
Jun 3 00:41:39 nas2 lxd.daemon[14615]: => Starting LXD
Jun 3 00:41:39 nas2 lxd.daemon[14615]: t=2023-06-03T00:41:39+0200 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored."
Jun 3 00:41:39 nas2 lxd.daemon[14615]: t=2023-06-03T00:41:39+0200 lvl=eror msg="Failed to start the daemon: Both legacy and new local database files exists"
Jun 3 00:41:39 nas2 lxd.daemon[14615]: Error: Both legacy and new local database files exists
Jun 3 00:41:40 nas2 lxd.daemon[14615]: => LXD failed to start
Jun 3 00:41:40 nas2 systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Jun 3 00:41:40 nas2 lxd.daemon[14732]: => Stop reason is: crashed
Jun 3 00:41:40 nas2 systemd[1]: snap.lxd.daemon.service: Unit entered failed state.
Jun 3 00:41:40 nas2 systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Jun 3 00:41:40 nas2 systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Jun 3 00:41:40 nas2 systemd[1]: Stopped Service for snap application lxd.daemon.
Jun 3 00:41:40 nas2 systemd[1]: Started Service for snap application lxd.daemon.
Jun 3 00:41:40 nas2 lxd.daemon[14764]: => Preparing the system
Jun 3 00:41:40 nas2 lxd.daemon[14764]: ==> Loading snap configuration
Jun 3 00:41:40 nas2 lxd.daemon[14764]: ==> Setting up mntns symlink (mnt:[4026532238])
Jun 3 00:41:40 nas2 lxd.daemon[14764]: ==> Setting up kmod wrapper
Jun 3 00:41:40 nas2 lxd.daemon[14764]: ==> Preparing /boot
Jun 3 00:41:40 nas2 lxd.daemon[14764]: ==> Preparing a clean copy of /run
Jun 3 00:41:40 nas2 lxd.daemon[14764]: ==> Preparing a clean copy of /etc
Jun 3 00:41:40 nas2 lxd.daemon[14764]: ==> Setting up ceph configuration
Jun 3 00:41:40 nas2 lxd.daemon[14764]: ==> Setting up LVM configuration
Jun 3 00:41:40 nas2 lxd.daemon[14764]: ==> Rotating logs
Jun 3 00:41:40 nas2 lxd.daemon[14764]: ==> Setting up ZFS (0.6)
Jun 3 00:41:40 nas2 lxd.daemon[14764]: ==> Escaping the systemd cgroups
Jun 3 00:41:40 nas2 lxd.daemon[14764]: ==> Escaping the systemd process resource limits
Jun 3 00:41:40 nas2 lxd.daemon[14306]: mount namespace: 7
Jun 3 00:41:40 nas2 lxd.daemon[14306]: hierarchies:
Jun 3 00:41:40 nas2 lxd.daemon[14306]: 0: fd: 8: blkio
Jun 3 00:41:40 nas2 lxd.daemon[14306]: 1: fd: 9: hugetlb
Jun 3 00:41:40 nas2 lxd.daemon[14306]: 2: fd: 10: pids
Jun 3 00:41:40 nas2 lxd.daemon[14306]: 3: fd: 11: devices
Jun 3 00:41:40 nas2 lxd.daemon[14306]: 4: fd: 12: net_cls,net_prio
Jun 3 00:41:40 nas2 lxd.daemon[14306]: 5: fd: 13: memory
Jun 3 00:41:40 nas2 lxd.daemon[14306]: 6: fd: 14: cpuset
Jun 3 00:41:40 nas2 lxd.daemon[14306]: 7: fd: 15: perf_event
Jun 3 00:41:40 nas2 lxd.daemon[14306]: 8: fd: 16: cpu,cpuacct
Jun 3 00:41:40 nas2 lxd.daemon[14306]: 9: fd: 17: freezer
Jun 3 00:41:40 nas2 lxd.daemon[14306]: 10: fd: 18: name=systemd
Jun 3 00:41:40 nas2 lxd.daemon[14306]: lxcfs.c: 105: do_reload: lxcfs: reloaded
Jun 3 00:41:40 nas2 lxd.daemon[14764]: => Re-using existing LXCFS
Jun 3 00:41:41 nas2 lxd.daemon[14764]: => Starting LXD
Jun 3 00:41:41 nas2 lxd.daemon[14764]: t=2023-06-03T00:41:41+0200 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored."
Jun 3 00:41:41 nas2 lxd.daemon[14764]: t=2023-06-03T00:41:41+0200 lvl=eror msg="Failed to start the daemon: Both legacy and new local database files exists"
Jun 3 00:41:41 nas2 lxd.daemon[14764]: Error: Both legacy and new local database files exists
Jun 3 00:41:42 nas2 lxd.daemon[14764]: => LXD failed to start
Jun 3 00:41:42 nas2 systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Jun 3 00:41:42 nas2 lxd.daemon[14882]: => Stop reason is: crashed
Jun 3 00:41:42 nas2 systemd[1]: snap.lxd.daemon.service: Unit entered failed state.
Jun 3 00:41:42 nas2 systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Jun 3 00:41:42 nas2 systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Jun 3 00:41:42 nas2 systemd[1]: Stopped Service for snap application lxd.daemon.
Jun 3 00:41:42 nas2 systemd[1]: Started Service for snap application lxd.daemon.
Jun 3 00:41:42 nas2 lxd.daemon[14916]: => Preparing the system
Jun 3 00:41:42 nas2 lxd.daemon[14916]: ==> Loading snap configuration
Jun 3 00:41:42 nas2 lxd.daemon[14916]: ==> Setting up mntns symlink (mnt:[4026532238])
Jun 3 00:41:42 nas2 lxd.daemon[14916]: ==> Setting up kmod wrapper
Jun 3 00:41:42 nas2 lxd.daemon[14916]: ==> Preparing /boot
Jun 3 00:41:42 nas2 lxd.daemon[14916]: ==> Preparing a clean copy of /run
Jun 3 00:41:42 nas2 lxd.daemon[14916]: ==> Preparing a clean copy of /etc
Jun 3 00:41:42 nas2 lxd.daemon[14916]: ==> Setting up ceph configuration
Jun 3 00:41:42 nas2 lxd.daemon[14916]: ==> Setting up LVM configuration
Jun 3 00:41:42 nas2 lxd.daemon[14916]: ==> Rotating logs
Jun 3 00:41:42 nas2 lxd.daemon[14916]: ==> Setting up ZFS (0.6)
Jun 3 00:41:42 nas2 lxd.daemon[14916]: ==> Escaping the systemd cgroups
Jun 3 00:41:42 nas2 lxd.daemon[14916]: ==> Escaping the systemd process resource limits
Jun 3 00:41:42 nas2 lxd.daemon[14306]: mount namespace: 7
Jun 3 00:41:42 nas2 lxd.daemon[14306]: hierarchies:
Jun 3 00:41:42 nas2 lxd.daemon[14306]: 0: fd: 8: blkio
Jun 3 00:41:42 nas2 lxd.daemon[14306]: 1: fd: 9: hugetlb
Jun 3 00:41:42 nas2 lxd.daemon[14306]: 2: fd: 10: pids
Jun 3 00:41:42 nas2 lxd.daemon[14306]: 3: fd: 11: devices
Jun 3 00:41:42 nas2 lxd.daemon[14306]: 4: fd: 12: net_cls,net_prio
Jun 3 00:41:42 nas2 lxd.daemon[14306]: 5: fd: 13: memory
Jun 3 00:41:42 nas2 lxd.daemon[14306]: 6: fd: 14: cpuset
Jun 3 00:41:42 nas2 lxd.daemon[14306]: 7: fd: 15: perf_event
Jun 3 00:41:42 nas2 lxd.daemon[14306]: 8: fd: 16: cpu,cpuacct
Jun 3 00:41:42 nas2 lxd.daemon[14306]: 9: fd: 17: freezer
Jun 3 00:41:42 nas2 lxd.daemon[14306]: 10: fd: 18: name=systemd
Jun 3 00:41:42 nas2 lxd.daemon[14306]: lxcfs.c: 105: do_reload: lxcfs: reloaded
Jun 3 00:41:42 nas2 lxd.daemon[14916]: => Re-using existing LXCFS
Jun 3 00:41:42 nas2 lxd.daemon[14916]: => Starting LXD
Jun 3 00:41:42 nas2 lxd.daemon[14916]: t=2023-06-03T00:41:42+0200 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored."
Jun 3 00:41:42 nas2 lxd.daemon[14916]: t=2023-06-03T00:41:42+0200 lvl=eror msg="Failed to start the daemon: Both legacy and new local database files exists"
Jun 3 00:41:42 nas2 lxd.daemon[14916]: Error: Both legacy and new local database files exists
Jun 3 00:41:43 nas2 lxd.daemon[14916]: => LXD failed to start
Jun 3 00:41:43 nas2 systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Jun 3 00:41:43 nas2 lxd.daemon[15036]: => Stop reason is: crashed
Jun 3 00:41:43 nas2 systemd[1]: snap.lxd.daemon.service: Unit entered failed state.
Jun 3 00:41:43 nas2 systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Jun 3 00:41:44 nas2 systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Jun 3 00:41:44 nas2 systemd[1]: Stopped Service for snap application lxd.daemon.
Jun 3 00:41:44 nas2 systemd[1]: snap.lxd.daemon.service: Start request repeated too quickly.
Jun 3 00:41:44 nas2 systemd[1]: Failed to start Service for snap application lxd.daemon.
Jun 3 00:41:44 nas2 systemd[1]: snap.lxd.daemon.service: Unit entered failed state.
Jun 3 00:41:44 nas2 systemd[1]: snap.lxd.daemon.service: Failed with result 'start-limit-hit'.
So while I can see that the snap.lxd.daemon.service: Failed with result ‘start-limit-hit’ I’m not sure why this happens or how I should proceed.
Any ideas on what I can try next ?
The problem is:
Jun 3 00:41:42 nas2 lxd.daemon[14916]: Error: Both legacy and new local database files exists
But I’m not sure why lxd.migrate didn’t work here.
You didn’t do lxd init before lxd.migrate did you?
I found https://github.com/lxc/lxd/issues/4542 which may help.
And LXD v3, Failed to start the daemon: Both legacy and new local database files exists - #6 by stgraber
Hi Thomas, thanks for looking into this.
Right, I did not do an lxd init.
I saw the Error: Both legacy and new local database files exists error too. But I see only the new database :
Old one seems gone :
20230606_232048 root@nas2:~/ ls -la /var/lib/lxd/
ls: cannot access '/var/lib/lxd/': No such file or directory
New tree :
20230606_232306 root@nas2:~/ ls -la /var/snap/lxd/common/lxd/
total 236
drwxr-xr-x 16 root root 4096 Jun 3 00:41 .
drwxr-xr-x 7 root root 4096 Jun 3 00:41 ..
drwx------ 2 root root 4096 Jun 3 00:41 cache
drwx--x--x 16 root root 4096 Jun 3 00:41 containers
drwx------ 3 root root 4096 Sep 3 2018 database
drwx------ 3 root root 4096 Jun 3 00:41 database.pre-migration
drwx--x--x 13 root root 4096 May 31 16:22 devices
drwxr-xr-x 2 root root 4096 Sep 3 2018 devlxd
drwx------ 2 root root 4096 Sep 3 2018 disks
drwx------ 2 root root 4096 Sep 3 2018 images
drwxr-xr-x 2 root root 4096 Jun 3 00:41 logs
-rw-r--r-- 1 root root 78848 Jun 3 00:41 lxd.db
-rw-rw-r-- 1 root root 78848 Jun 3 00:41 lxd.db.pre-migration
drwx--x--x 2 root root 4096 Sep 3 2018 networks
drwx------ 4 root root 4096 Sep 5 2018 security
-rw-r--r-- 1 root root 1899 Sep 3 2018 server.crt
-rw------- 1 root root 3243 Sep 3 2018 server.key
drwx--x--x 2 root root 4096 Sep 3 2018 shmounts
drwx------ 13 root root 4096 May 31 16:22 snapshots
drwx--x--x 3 root root 4096 Sep 4 2018 storage-pools
srw-rw---- 1 root lxd 0 Sep 22 2021 unix.socket
Is it perhaps complaining about some of these ?
20230606_232724 root@nas2:~/ find /var/snap/lxd/common/ -iname "*db"
/var/snap/lxd/common/lxd/lxd.db
/var/snap/lxd/common/lxd/database.pre-migration/global/logs.db
/var/snap/lxd/common/lxd/database.pre-migration/local.db
/var/snap/lxd/common/lxd/database/global/logs.db
/var/snap/lxd/common/lxd/database/local.db
About https://github.com/lxc/lxd/issues/4542 :
There are no snap changes
20230606_232757 root@nas2:~/ snap changes
no changes found
But there is that db file
20230606_232900 root@nas2:~/ l /var/snap/lxd/common/lxd/lxd.db
-rw-r--r-- 1 root root 78848 Jun 3 00:41 /var/snap/lxd/common/lxd/lxd.db
About LXD v3, Failed to start the daemon: Both legacy and new local database files exists - #6 by stgraber
There is no raft in that location :
20230606_233157 root@nas2:~/ l /var/snap/lxd/common/raft
ls: cannot access '/var/snap/lxd/common/raft': No such file or directory
and the /var/snap/lxd/common/lxd.db does not exist either, it is located in a different location at /var/snap/lxd/common/lxd/lxd.db .
Can I safely move that /var/snap/lxd/common/lxd/lxd.db aside and use systemctl restart snap.lxd.daemon to see if the still running /snap/bin/lxd.migrate could then proceed ?
Yes I would try that, keep a copy of it, but its not a standard location file for the DBs in current versions.
Update for others who might have the same issue: Moving /var/snap/lxd/common/lxd/lxd.db aside allowed the new snap.lxd.daemon to start and the migrate process to ‘complete’. However nothing was migrated properly as the new snap.lxd.daemon only listed some old test containers from 5 years ago (2018).