atrius
April 8, 2020, 10:56pm
1
Twice I’ve tried to use lxd.migrate to migrate a system from 3.0.3 to 4.0.0 and twice it failed horribly, leaving the system in an initially unusable state.
However, this second time I poked around a bit. The initial system was that lxc list and all other similar commands such as lxc network list all came back blank. This told me that the database had been wiped for 3.0.3. I found the backup though in /var/snap/lxd/common/lxd/database.pre-migration. I copied them from there to /var/lib/lxd/database and was able to bring things back online.
Is it normal for lxd.migrate to fail in such an ungraceful manner?
atrius
April 8, 2020, 11:46pm
2
Recovered the system. Backed everything up again just to be sure. Gave lxd.migrate another chance… same result. Borked the system. This time, it just hung at “Waiting for LXD to come online”. Syslog indicates it is just spinning, starting and restating over and over again
stgraber
April 9, 2020, 2:03am
3
We do have quite a lot of automated testing on lxd.migrate, however every system is different and issues can happen. The migration logic does save enough data for us to figure out what may have gone wrong and unstick it.
Can you paste the entire output of your lxd.migrate run?
Also the output of journalctl -u snap.lxd.daemon -n 300 and the content of /var/snap/lxd/common/lxd/logs/lxd.log would be quite useful.
atrius
April 9, 2020, 4:43am
4
I’ll see what I can find. I needed the system up so just kinda worked around it haha
If memory serves though, it got to the step where it was waiting for the new LXD to come up and just sat there forever. In syslog, it was spinning trying to start and failing. I don’t have the output from lxd.migrate now but do have the syslog entries:Apr 8 23:46:26 mayne systemd[1]: Started Service for snap application lxd.daemon. Apr 8 23:46:26 mayne lxd.daemon[29495]: => Preparing the system (14353) Apr 8 23:46:26 mayne lxd.daemon[29495]: ==> Loading snap configuration Apr 8 23:46:26 mayne lxd.daemon[29495]: ==> Setting up mntns symlink (mnt:[4026533028]) Apr 8 23:46:26 mayne lxd.daemon[29495]: ==> Setting up kmod wrapper Apr 8 23:46:26 mayne lxd.daemon[29495]: ==> Preparing /boot Apr 8 23:46:26 mayne lxd.daemon[29495]: ==> Preparing a clean copy of /run Apr 8 23:46:26 mayne lxd.daemon[29495]: ==> Preparing a clean copy of /etc Apr 8 23:46:26 mayne lxd.daemon[29495]: ==> Setting up ceph configuration Apr 8 23:46:26 mayne lxd.daemon[29495]: ==> Setting up LVM configuration Apr 8 23:46:26 mayne lxd.daemon[29495]: ==> Rotating logs Apr 8 23:46:26 mayne lxd.daemon[29495]: ==> Setting up ZFS (0.7) Apr 8 23:46:26 mayne lxd.daemon[29495]: ==> Escaping the systemd cgroups Apr 8 23:46:26 mayne lxd.daemon[29495]: ====> Detected cgroup V1 Apr 8 23:46:26 mayne lxd.daemon[29495]: ==> Escaping the systemd process resource limits Apr 8 23:46:26 mayne lxd.daemon[29495]: ==> Disabling shiftfs on this kernel (auto) Apr 8 23:46:26 mayne lxd.daemon[23436]: Closed liblxcfs.so Apr 8 23:46:26 mayne lxd.daemon[23436]: Running destructor lxcfs_exit Apr 8 23:46:26 mayne lxd.daemon[23436]: Running constructor lxcfs_init to reload liblxcfs Apr 8 23:46:26 mayne kernel: [ 5325.292617] new mount options do not match the existing superblock, will be ignored Apr 8 23:46:26 mayne lxd.daemon[23436]: mount namespace: 5 Apr 8 23:46:26 mayne lxd.daemon[23436]: hierarchies: Apr 8 23:46:26 mayne lxd.daemon[23436]: 0: fd: 6: Apr 8 23:46:26 mayne lxd.daemon[23436]: 1: fd: 7: name=systemd Apr 8 23:46:26 mayne lxd.daemon[23436]: 2: fd: 8: pids Apr 8 23:46:26 mayne lxd.daemon[23436]: 3: fd: 9: freezer Apr 8 23:46:26 mayne lxd.daemon[23436]: 4: fd: 10: net_cls,net_prio Apr 8 23:46:26 mayne lxd.daemon[23436]: 5: fd: 11: cpuset Apr 8 23:46:26 mayne lxd.daemon[23436]: 6: fd: 12: cpu,cpuacct Apr 8 23:46:26 mayne lxd.daemon[23436]: 7: fd: 13: rdma Apr 8 23:46:26 mayne lxd.daemon[23436]: 8: fd: 14: memory Apr 8 23:46:26 mayne lxd.daemon[23436]: 9: fd: 15: devices Apr 8 23:46:26 mayne lxd.daemon[23436]: 10: fd: 16: perf_event Apr 8 23:46:26 mayne lxd.daemon[23436]: 11: fd: 17: hugetlb Apr 8 23:46:26 mayne lxd.daemon[23436]: 12: fd: 19: blkio Apr 8 23:46:26 mayne lxd.daemon[23436]: api_extensions: Apr 8 23:46:26 mayne lxd.daemon[23436]: - cgroups Apr 8 23:46:26 mayne lxd.daemon[23436]: - sys_cpu_online Apr 8 23:46:26 mayne lxd.daemon[23436]: - proc_cpuinfo Apr 8 23:46:26 mayne lxd.daemon[23436]: - proc_diskstats Apr 8 23:46:26 mayne lxd.daemon[23436]: - proc_loadavg Apr 8 23:46:26 mayne lxd.daemon[23436]: - proc_meminfo Apr 8 23:46:26 mayne lxd.daemon[23436]: - proc_stat Apr 8 23:46:26 mayne lxd.daemon[23436]: - proc_swaps Apr 8 23:46:26 mayne lxd.daemon[23436]: - proc_uptime Apr 8 23:46:26 mayne lxd.daemon[23436]: - shared_pidns Apr 8 23:46:26 mayne lxd.daemon[23436]: - cpuview_daemon Apr 8 23:46:26 mayne lxd.daemon[23436]: - loadavg_daemon Apr 8 23:46:26 mayne lxd.daemon[23436]: - pidfds Apr 8 23:46:26 mayne lxd.daemon[23436]: Reloaded LXCFS Apr 8 23:46:26 mayne lxd.daemon[29495]: => Re-using existing LXCFS Apr 8 23:46:26 mayne lxd.daemon[29495]: => Starting LXD Apr 8 23:46:26 mayne lxd.daemon[29495]: t=2020-04-08T23:46:26+0000 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored" Apr 8 23:46:27 mayne lxd.daemon[29495]: *** Error in lxd’: munmap_chunk(): invalid pointer: 0x00007f34c032f090 ***
Not all failures were as verbose as that one. Most didn’t include the memory map. I actually just tried to do it again, however it said the source server was empty so no migration was needed. The LXD daemon was able to start this time so that’s something
stgraber
April 9, 2020, 2:06pm
5
@freeekanayaka dqlite crash looks like ^
