I have a cluster of 5 with ceph storage backbone. lxc stop responding this afternoon due to a version mismatch on some of the servers which failed to do the auto refresh. I then tried to refresh the servers manually by running
sudo snap refresh lxd --channel=4.24/stable
It stuck at
snap “lxd” start
Any help will be much appreciated please.
Terry
output of running
journalctl -u snap.lxd.daemon -n 300
– Logs begin at Fri 2020-11-27 19:45:41 HKT, end at Mon 2022-03-21 05:25:17 HKT. –
Mar 21 04:15:15 octo lxd.daemon[17510]: - proc_swaps
Mar 21 04:15:15 octo lxd.daemon[17510]: - proc_uptime
Mar 21 04:15:15 octo lxd.daemon[17510]: - shared_pidns
Mar 21 04:15:15 octo lxd.daemon[17510]: - cpuview_daemon
Mar 21 04:15:15 octo lxd.daemon[17510]: - loadavg_daemon
Mar 21 04:15:15 octo lxd.daemon[17510]: - pidfds
Mar 21 04:15:15 octo lxd.daemon[17510]: Reloaded LXCFS
Mar 21 04:15:15 octo lxd.daemon[24463]: => Starting LXD
Mar 21 04:15:15 octo lxd.daemon[24611]: t=2022-03-21T04:15:15+0800 lvl=warn msg=" - Couldn’t find the CGro>
Mar 21 04:15:15 octo lxd.daemon[24611]: t=2022-03-21T04:15:15+0800 lvl=warn msg=" - Couldn’t find the CGro>
Mar 21 04:15:16 octo lxd.daemon[24611]: t=2022-03-21T04:15:16+0800 lvl=warn msg="Dqlite: attempt 1: server>
Mar 21 04:15:16 octo lxd.daemon[24611]: t=2022-03-21T04:15:16+0800 lvl=warn msg="Dqlite: attempt 1: server>
Mar 21 04:15:16 octo lxd.daemon[24611]: t=2022-03-21T04:15:16+0800 lvl=warn msg="Dqlite: attempt 1: server>
Mar 21 04:15:16 octo lxd.daemon[24611]: t=2022-03-21T04:15:16+0800 lvl=eror msg=“Failed to start the daemo>
Mar 21 04:15:16 octo lxd.daemon[24611]: t=2022-03-21T04:15:16+0800 lvl=warn msg=“Could not handover member>
Mar 21 04:15:16 octo lxd.daemon[24611]: Error: Failed to initialize global database: failed to ensure sche>
Mar 21 04:15:16 octo lxd.daemon[24463]: => LXD failed to start
Mar 21 04:15:16 octo systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILU>
Mar 21 04:15:16 octo systemd[1]: snap.lxd.daemon.service: Failed with result ‘exit-code’.
Mar 21 04:15:16 octo systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 20.
Mar 21 04:15:16 octo systemd[1]: Stopped Service for snap application lxd.daemon.
Mar 21 04:15:16 octo systemd[1]: Started Service for snap application lxd.daemon.
Mar 21 04:15:16 octo lxd.daemon[24658]: => Preparing the system (22652)
Mar 21 04:15:16 octo lxd.daemon[24658]: ==> Loading snap configuration
Mar 21 04:15:16 octo lxd.daemon[24658]: ==> Setting up mntns symlink (mnt:[4026532600])
Mar 21 04:15:16 octo lxd.daemon[24658]: ==> Setting up kmod wrapper
Mar 21 04:15:16 octo lxd.daemon[24658]: ==> Preparing /boot
Mar 21 04:15:16 octo lxd.daemon[24658]: ==> Preparing a clean copy of /run
Mar 21 04:15:16 octo lxd.daemon[24658]: ==> Preparing /run/bin
Mar 21 04:15:16 octo lxd.daemon[24658]: ==> Preparing a clean copy of /etc
Mar 21 04:15:16 octo lxd.daemon[24658]: ==> Preparing a clean copy of /usr/share/misc
Mar 21 04:15:16 octo lxd.daemon[24658]: ==> Setting up ceph configuration
Mar 21 04:15:16 octo lxd.daemon[24658]: ==> Setting up LVM configuration
Mar 21 04:15:16 octo lxd.daemon[24658]: ==> Setting up OVN configuration
Mar 21 04:15:16 octo lxd.daemon[24658]: ==> Rotating logs
Mar 21 04:15:16 octo lxd.daemon[24658]: ==> Setting up ZFS (0.8)
Mar 21 04:15:16 octo lxd.daemon[24658]: ==> Escaping the systemd cgroups
Mar 21 04:15:16 octo lxd.daemon[24658]: ====> Detected cgroup V1
Mar 21 04:15:16 octo lxd.daemon[24658]: ==> Escaping the systemd process resource limits
Mar 21 04:15:16 octo lxd.daemon[24658]: ==> Disabling shiftfs on this kernel (auto)
Mar 21 04:15:16 octo lxd.daemon[17510]: Closed liblxcfs.so
Mar 21 04:15:16 octo lxd.daemon[17510]: Running destructor lxcfs_exit
Mar 21 04:15:16 octo lxd.daemon[17510]: Running constructor lxcfs_init to reload liblxcfs
Mar 21 04:15:16 octo lxd.daemon[17510]: mount namespace: 5
Mar 21 04:15:16 octo lxd.daemon[17510]: hierarchies:
Mar 21 04:15:16 octo lxd.daemon[17510]: 0: fd: 6:
Mar 21 04:15:16 octo lxd.daemon[17510]: 1: fd: 7: name=systemd
Mar 21 04:15:16 octo lxd.daemon[17510]: 2: fd: 8: cpuset
Mar 21 04:15:16 octo lxd.daemon[17510]: 3: fd: 9: blkio
Mar 21 04:15:16 octo lxd.daemon[17510]: 4: fd: 10: memory
Mar 21 04:15:16 octo lxd.daemon[17510]: 5: fd: 11: cpu,cpuacct
Mar 21 04:15:16 octo lxd.daemon[17510]: 6: fd: 12: devices
Mar 21 04:15:16 octo lxd.daemon[17510]: 7: fd: 13: rdma
Mar 21 04:15:16 octo lxd.daemon[17510]: 8: fd: 14: pids
Mar 21 04:15:16 octo lxd.daemon[17510]: 9: fd: 15: net_cls,net_prio
Mar 21 04:15:16 octo lxd.daemon[17510]: 10: fd: 16: freezer
Mar 21 04:15:16 octo lxd.daemon[17510]: 11: fd: 17: perf_event
Mar 21 04:15:16 octo lxd.daemon[17510]: 12: fd: 19: hugetlb
Mar 21 04:15:16 octo lxd.daemon[17510]: Kernel supports pidfds
Mar 21 04:15:16 octo lxd.daemon[17510]: Kernel does not support swap accounting
Mar 21 04:15:16 octo lxd.daemon[17510]: api_extensions:
Mar 21 04:15:16 octo lxd.daemon[17510]: - cgroups
Mar 21 04:15:16 octo lxd.daemon[17510]: - sys_cpu_online
Mar 21 04:15:16 octo lxd.daemon[17510]: - proc_cpuinfo
Mar 21 04:15:16 octo lxd.daemon[17510]: - proc_diskstats
Mar 21 04:15:16 octo lxd.daemon[17510]: - proc_loadavg
Mar 21 04:15:16 octo lxd.daemon[17510]: - proc_meminfo
Mar 21 04:15:16 octo lxd.daemon[17510]: - proc_stat
Mar 21 04:15:16 octo lxd.daemon[17510]: - proc_swaps
Mar 21 04:15:16 octo lxd.daemon[17510]: - proc_uptime
Mar 21 04:15:16 octo lxd.daemon[17510]: - shared_pidns
Mar 21 04:15:16 octo lxd.daemon[17510]: - cpuview_daemon
Mar 21 04:15:16 octo lxd.daemon[17510]: - loadavg_daemon
Mar 21 04:15:16 octo lxd.daemon[17510]: - pidfds
Mar 21 04:15:16 octo lxd.daemon[17510]: Reloaded LXCFS
Mar 21 04:15:16 octo lxd.daemon[24658]: => Re-using existing LXCFS
Mar 21 04:15:16 octo lxd.daemon[24658]: => Starting LXD
Mar 21 04:15:17 octo lxd.daemon[24807]: t=2022-03-21T04:15:17+0800 lvl=warn msg=” - Couldn’t find the CGro>
Mar 21 04:15:17 octo lxd.daemon[24807]: t=2022-03-21T04:15:17+0800 lvl=warn msg=” - Couldn’t find the CGro>
Mar 21 04:15:17 octo lxd.daemon[24807]: t=2022-03-21T04:15:17+0800 lvl=warn msg="Dqlite: attempt 1: server>
Mar 21 04:15:17 octo lxd.daemon[24807]: t=2022-03-21T04:15:17+0800 lvl=warn msg="Dqlite: attempt 1: server>
Mar 21 04:15:17 octo lxd.daemon[24807]: t=2022-03-21T04:15:17+0800 lvl=warn msg="Dqlite: attempt 1: server>
Mar 21 04:15:17 octo lxd.daemon[24807]: t=2022-03-21T04:15:17+0800 lvl=eror msg=“Failed to start the daemo>
Mar 21 04:15:17 octo lxd.daemon[24807]: t=2022-03-21T04:15:17+0800 lvl=warn msg=“Could not handover member>
Mar 21 04:15:17 octo lxd.daemon[24807]: Error: Failed to initialize global database: failed to ensure sche>
Mar 21 04:15:17 octo lxd.daemon[24658]: => LXD failed to start
Mar 21 04:15:17 octo systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILU>
Mar 21 04:15:17 octo systemd[1]: snap.lxd.daemon.service: Failed with result ‘exit-code’.
Mar 21 04:15:18 octo systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 21.
Mar 21 04:15:18 octo systemd[1]: Stopped Service for snap application lxd.daemon.
Mar 21 04:15:18 octo systemd[1]: Started Service for snap application lxd.daemon.
Mar 21 04:15:18 octo lxd.daemon[24856]: => Preparing the system (22652)
Mar 21 04:15:18 octo lxd.daemon[24856]: ==> Loading snap configuration
Mar 21 04:15:18 octo lxd.daemon[24856]: ==> Setting up mntns symlink (mnt:[4026532600])
Mar 21 04:15:18 octo lxd.daemon[24856]: ==> Setting up kmod wrapper
Mar 21 04:15:18 octo lxd.daemon[24856]: ==> Preparing /boot
Mar 21 04:15:18 octo lxd.daemon[24856]: ==> Preparing a clean copy of /run
Mar 21 04:15:18 octo lxd.daemon[24856]: ==> Preparing /run/bin
Mar 21 04:15:18 octo lxd.daemon[24856]: ==> Preparing a clean copy of /etc
Mar 21 04:15:18 octo lxd.daemon[24856]: ==> Preparing a clean copy of /usr/share/misc
Mar 21 04:15:18 octo lxd.daemon[24856]: ==> Setting up ceph configuration
Mar 21 04:15:18 octo lxd.daemon[24856]: ==> Setting up LVM configuration
Mar 21 04:15:18 octo lxd.daemon[24856]: ==> Setting up OVN configuration
Mar 21 04:15:18 octo lxd.daemon[24856]: ==> Rotating logs
Mar 21 04:15:19 octo lxd.daemon[24856]: ==> Setting up ZFS (0.8)
Mar 21 04:15:19 octo lxd.daemon[24856]: ==> Escaping the systemd cgroups
Mar 21 04:15:19 octo lxd.daemon[24856]: ====> Detected cgroup V1
Mar 21 04:15:19 octo lxd.daemon[24856]: ==> Escaping the systemd process resource limits
Mar 21 04:15:19 octo lxd.daemon[24856]: ==> Disabling shiftfs on this kernel (auto)
Mar 21 04:15:19 octo lxd.daemon[17510]: Closed liblxcfs.so
Mar 21 04:15:19 octo lxd.daemon[17510]: Running destructor lxcfs_exit
Mar 21 04:15:19 octo lxd.daemon[17510]: Running constructor lxcfs_init to reload liblxcfs
Mar 21 04:15:19 octo lxd.daemon[17510]: mount namespace: 5
Mar 21 04:15:19 octo lxd.daemon[17510]: hierarchies:
Mar 21 04:15:19 octo lxd.daemon[17510]: 0: fd: 6:
Mar 21 04:15:19 octo lxd.daemon[17510]: 1: fd: 7: name=systemd
Mar 21 04:15:19 octo lxd.daemon[17510]: 2: fd: 8: cpuset
Mar 21 04:15:19 octo lxd.daemon[17510]: 3: fd: 9: blkio
Mar 21 04:15:19 octo lxd.daemon[17510]: 4: fd: 10: memory
Mar 21 04:15:19 octo lxd.daemon[17510]: 5: fd: 11: cpu,cpuacct
Mar 21 04:15:19 octo lxd.daemon[17510]: 6: fd: 12: devices
Mar 21 04:15:19 octo lxd.daemon[17510]: 7: fd: 13: rdma
Mar 21 04:15:19 octo lxd.daemon[17510]: 8: fd: 14: pids
Mar 21 04:15:19 octo lxd.daemon[17510]: 9: fd: 15: net_cls,net_prio
Mar 21 04:15:19 octo lxd.daemon[17510]: 10: fd: 16: freezer
Mar 21 04:15:19 octo lxd.daemon[17510]: 11: fd: 17: perf_event
Mar 21 04:15:19 octo lxd.daemon[17510]: 12: fd: 19: hugetlb
Mar 21 04:15:19 octo lxd.daemon[17510]: Kernel supports pidfds
Mar 21 04:15:19 octo lxd.daemon[17510]: Kernel does not support swap accounting
Mar 21 04:15:19 octo lxd.daemon[17510]: api_extensions:
Mar 21 04:15:19 octo lxd.daemon[17510]: - cgroups
Mar 21 04:15:19 octo lxd.daemon[17510]: - sys_cpu_online
Mar 21 04:15:19 octo lxd.daemon[17510]: - proc_cpuinfo
Mar 21 04:15:19 octo lxd.daemon[17510]: - proc_diskstats
Mar 21 04:15:19 octo lxd.daemon[17510]: - proc_loadavg
Mar 21 04:15:19 octo lxd.daemon[17510]: - proc_meminfo
Mar 21 04:15:19 octo lxd.daemon[17510]: - proc_stat
Mar 21 04:15:19 octo lxd.daemon[17510]: - proc_swaps
Mar 21 04:15:19 octo lxd.daemon[17510]: - proc_uptime
Mar 21 04:15:19 octo lxd.daemon[17510]: - shared_pidns
Mar 21 04:15:19 octo lxd.daemon[17510]: - cpuview_daemon
Mar 21 04:15:19 octo lxd.daemon[17510]: - loadavg_daemon
Mar 21 04:15:19 octo lxd.daemon[17510]: - pidfds
Mar 21 04:15:19 octo lxd.daemon[17510]: Reloaded LXCFS
Mar 21 04:15:19 octo lxd.daemon[24856]: => Re-using existing LXCFS
Mar 21 04:15:19 octo lxd.daemon[24856]: => Starting LXD
Mar 21 04:15:19 octo lxd.daemon[25004]: t=2022-03-21T04:15:19+0800 lvl=warn msg=” - Couldn’t find the CGro>
Mar 21 04:15:19 octo lxd.daemon[25004]: t=2022-03-21T04:15:19+0800 lvl=warn msg=” - Couldn’t find the CGro>
Mar 21 04:15:19 octo lxd.daemon[25004]: t=2022-03-21T04:15:19+0800 lvl=warn msg="Dqlite: attempt 1: server>
Mar 21 04:15:19 octo lxd.daemon[25004]: t=2022-03-21T04:15:19+0800 lvl=warn msg="Dqlite: attempt 1: server>
Mar 21 04:15:19 octo lxd.daemon[25004]: t=2022-03-21T04:15:19+0800 lvl=warn msg="Dqlite: attempt 1: server>
Mar 21 04:15:20 octo lxd.daemon[25004]: t=2022-03-21T04:15:20+0800 lvl=eror msg=“Failed to start the daemo>
Mar 21 04:15:21 octo lxd.daemon[25004]: t=2022-03-21T04:15:21+0800 lvl=warn msg=“Could not handover member>
Mar 21 04:15:21 octo lxd.daemon[25004]: Error: Failed to initialize global database: failed to ensure sche>
Mar 21 04:15:21 octo lxd.daemon[24856]: => LXD failed to start
Mar 21 04:15:21 octo systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILU>
Mar 21 04:15:21 octo systemd[1]: snap.lxd.daemon.service: Failed with result ‘exit-code’.
Mar 21 04:15:21 octo systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 22.
Mar 21 04:15:21 octo systemd[1]: Stopped Service for snap application lxd.daemon.
Mar 21 04:15:21 octo systemd[1]: Started Service for snap application lxd.daemon.
Mar 21 04:15:21 octo lxd.daemon[25057]: => Preparing the system (22652)
Mar 21 04:15:21 octo lxd.daemon[25057]: ==> Loading snap configuration
Mar 21 04:15:21 octo lxd.daemon[25057]: ==> Setting up mntns symlink (mnt:[4026532600])
Mar 21 04:15:21 octo lxd.daemon[25057]: ==> Setting up kmod wrapper
Mar 21 04:15:21 octo lxd.daemon[25057]: ==> Preparing /boot
Mar 21 04:15:21 octo lxd.daemon[25057]: ==> Preparing a clean copy of /run
Mar 21 04:15:21 octo lxd.daemon[25057]: ==> Preparing /run/bin
Mar 21 04:15:21 octo lxd.daemon[25057]: ==> Preparing a clean copy of /etc
Mar 21 04:15:21 octo lxd.daemon[25057]: ==> Preparing a clean copy of /usr/share/misc
Mar 21 04:15:21 octo lxd.daemon[25057]: ==> Setting up ceph configuration
Mar 21 04:15:21 octo lxd.daemon[25057]: ==> Setting up LVM configuration
Mar 21 04:15:21 octo lxd.daemon[25057]: ==> Setting up OVN configuration
Mar 21 04:15:21 octo lxd.daemon[25057]: ==> Rotating logs
Mar 21 04:15:21 octo lxd.daemon[25057]: ==> Setting up ZFS (0.8)
Mar 21 04:15:21 octo lxd.daemon[25057]: ==> Escaping the systemd cgroups
Mar 21 04:15:21 octo lxd.daemon[25057]: ====> Detected cgroup V1
Mar 21 04:15:21 octo lxd.daemon[25057]: ==> Escaping the systemd process resource limits
Mar 21 04:15:21 octo lxd.daemon[25057]: ==> Disabling shiftfs on this kernel (auto)
Mar 21 04:15:21 octo lxd.daemon[17510]: Closed liblxcfs.so
Mar 21 04:15:21 octo lxd.daemon[17510]: Running destructor lxcfs_exit
Mar 21 04:15:21 octo lxd.daemon[17510]: Running constructor lxcfs_init to reload liblxcfs
Mar 21 04:15:21 octo lxd.daemon[17510]: mount namespace: 5
Mar 21 04:15:21 octo lxd.daemon[17510]: hierarchies:
Mar 21 04:15:21 octo lxd.daemon[17510]: 0: fd: 6:
Mar 21 04:15:21 octo lxd.daemon[17510]: 1: fd: 7: name=systemd
Mar 21 04:15:21 octo lxd.daemon[17510]: 2: fd: 8: cpuset
Mar 21 04:15:21 octo lxd.daemon[17510]: 3: fd: 9: blkio
Mar 21 04:15:21 octo lxd.daemon[17510]: 4: fd: 10: memory
Mar 21 04:15:21 octo lxd.daemon[17510]: 5: fd: 11: cpu,cpuacct
Mar 21 04:15:21 octo lxd.daemon[17510]: 6: fd: 12: devices
Mar 21 04:15:21 octo lxd.daemon[17510]: 7: fd: 13: rdma
Mar 21 04:15:21 octo lxd.daemon[17510]: 8: fd: 14: pids
Mar 21 04:15:21 octo lxd.daemon[17510]: 9: fd: 15: net_cls,net_prio
Mar 21 04:15:21 octo lxd.daemon[17510]: 10: fd: 16: freezer
Mar 21 04:15:21 octo lxd.daemon[17510]: 11: fd: 17: perf_event
Mar 21 04:15:21 octo lxd.daemon[17510]: 12: fd: 19: hugetlb
Mar 21 04:15:21 octo lxd.daemon[17510]: Kernel supports pidfds
Mar 21 04:15:21 octo lxd.daemon[17510]: Kernel does not support swap accounting
Mar 21 04:15:21 octo lxd.daemon[17510]: api_extensions:
Mar 21 04:15:21 octo lxd.daemon[17510]: - cgroups
Mar 21 04:15:21 octo lxd.daemon[17510]: - sys_cpu_online
Mar 21 04:15:21 octo lxd.daemon[17510]: - proc_cpuinfo
Mar 21 04:15:21 octo lxd.daemon[17510]: - proc_diskstats
Mar 21 04:15:21 octo lxd.daemon[17510]: - proc_loadavg
Mar 21 04:15:21 octo lxd.daemon[17510]: - proc_meminfo
Mar 21 04:15:21 octo lxd.daemon[17510]: - proc_stat
Mar 21 04:15:21 octo lxd.daemon[17510]: - proc_swaps
Mar 21 04:15:21 octo lxd.daemon[17510]: - proc_uptime
Mar 21 04:15:21 octo lxd.daemon[17510]: - shared_pidns
Mar 21 04:15:21 octo lxd.daemon[17510]: - cpuview_daemon
Mar 21 04:15:21 octo lxd.daemon[17510]: - loadavg_daemon
Mar 21 04:15:21 octo lxd.daemon[17510]: - pidfds
Mar 21 04:15:21 octo lxd.daemon[17510]: Reloaded LXCFS
Mar 21 04:15:21 octo lxd.daemon[25057]: => Re-using existing LXCFS
Mar 21 04:15:21 octo lxd.daemon[25057]: => Starting LXD
Mar 21 04:15:22 octo lxd.daemon[25205]: t=2022-03-21T04:15:22+0800 lvl=warn msg=” - Couldn’t find the CGro>
Mar 21 04:15:22 octo lxd.daemon[25205]: t=2022-03-21T04:15:22+0800 lvl=warn msg=” - Couldn’t find the CGro>
Mar 21 04:15:22 octo lxd.daemon[25205]: t=2022-03-21T04:15:22+0800 lvl=warn msg="Dqlite: attempt 1: server>
Mar 21 04:15:22 octo lxd.daemon[25205]: t=2022-03-21T04:15:22+0800 lvl=warn msg="Dqlite: attempt 1: server>
Mar 21 04:15:22 octo lxd.daemon[25205]: t=2022-03-21T04:15:22+0800 lvl=warn msg="Dqlite: attempt 1: server>
Mar 21 04:15:22 octo lxd.daemon[25205]: t=2022-03-21T04:15:22+0800 lvl=eror msg=“Failed to start the daemo>
Mar 21 04:15:22 octo lxd.daemon[25205]: t=2022-03-21T04:15:22+0800 lvl=warn msg=“Could not handover member>
Mar 21 04:15:22 octo lxd.daemon[25205]: Error: Failed to initialize global database: failed to ensure sche>
Mar 21 04:15:22 octo lxd.daemon[25057]: => LXD failed to start
Mar 21 04:15:22 octo systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILU>
Mar 21 04:15:22 octo systemd[1]: snap.lxd.daemon.service: Failed with result ‘exit-code’.
Mar 21 04:15:23 octo systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 23.
Mar 21 04:15:23 octo systemd[1]: Stopped Service for snap application lxd.daemon.
Mar 21 04:15:23 octo systemd[1]: Started Service for snap application lxd.daemon.
Mar 21 04:15:23 octo lxd.daemon[25258]: => Preparing the system (22652)
Mar 21 04:15:23 octo lxd.daemon[25258]: ==> Loading snap configuration
Mar 21 04:15:23 octo lxd.daemon[25258]: ==> Setting up mntns symlink (mnt:[4026532600])
Mar 21 04:15:23 octo lxd.daemon[25258]: ==> Setting up kmod wrapper
Mar 21 04:15:23 octo lxd.daemon[25258]: ==> Preparing /boot
Mar 21 04:15:23 octo lxd.daemon[25258]: ==> Preparing a clean copy of /run
Mar 21 04:15:23 octo lxd.daemon[25258]: ==> Preparing /run/bin
Mar 21 04:15:23 octo lxd.daemon[25258]: ==> Preparing a clean copy of /etc
Mar 21 04:15:23 octo lxd.daemon[25258]: ==> Preparing a clean copy of /usr/share/misc
Mar 21 04:15:23 octo lxd.daemon[25258]: ==> Setting up ceph configuration
Mar 21 04:15:23 octo lxd.daemon[25258]: ==> Setting up LVM configuration
Mar 21 04:15:23 octo lxd.daemon[25258]: ==> Setting up OVN configuration
Mar 21 04:15:23 octo lxd.daemon[25258]: ==> Rotating logs
Mar 21 04:15:23 octo lxd.daemon[25258]: ==> Setting up ZFS (0.8)
Mar 21 04:15:23 octo lxd.daemon[25258]: ==> Escaping the systemd cgroups
Mar 21 04:15:23 octo lxd.daemon[25258]: ====> Detected cgroup V1
Mar 21 04:15:23 octo lxd.daemon[25258]: ==> Escaping the systemd process resource limits
Mar 21 04:15:23 octo lxd.daemon[25258]: ==> Disabling shiftfs on this kernel (auto)
Mar 21 04:15:23 octo lxd.daemon[17510]: Closed liblxcfs.so
Mar 21 04:15:23 octo lxd.daemon[17510]: Running destructor lxcfs_exit
Mar 21 04:15:23 octo lxd.daemon[17510]: Running constructor lxcfs_init to reload liblxcfs
Mar 21 04:15:23 octo lxd.daemon[17510]: mount namespace: 5
Mar 21 04:15:23 octo lxd.daemon[17510]: hierarchies:
Mar 21 04:15:23 octo lxd.daemon[17510]: 0: fd: 6:
Mar 21 04:15:23 octo lxd.daemon[17510]: 1: fd: 7: name=systemd
Mar 21 04:15:23 octo lxd.daemon[17510]: 2: fd: 8: cpuset
Mar 21 04:15:23 octo lxd.daemon[17510]: 3: fd: 9: blkio
Mar 21 04:15:23 octo lxd.daemon[17510]: 4: fd: 10: memory
Mar 21 04:15:23 octo lxd.daemon[17510]: 5: fd: 11: cpu,cpuacct
Mar 21 04:15:23 octo lxd.daemon[17510]: 6: fd: 12: devices
Mar 21 04:15:23 octo lxd.daemon[17510]: 7: fd: 13: rdma
Mar 21 04:15:23 octo lxd.daemon[17510]: 8: fd: 14: pids
Mar 21 04:15:23 octo lxd.daemon[17510]: 9: fd: 15: net_cls,net_prio
Mar 21 04:15:23 octo lxd.daemon[17510]: 10: fd: 16: freezer
Mar 21 04:15:23 octo lxd.daemon[17510]: 11: fd: 17: perf_event
Mar 21 04:15:23 octo lxd.daemon[17510]: 12: fd: 19: hugetlb
Mar 21 04:15:23 octo lxd.daemon[17510]: Kernel supports pidfds
Mar 21 04:15:23 octo lxd.daemon[17510]: Kernel does not support swap accounting
Mar 21 04:15:23 octo lxd.daemon[17510]: api_extensions:
Mar 21 04:15:23 octo lxd.daemon[17510]: - cgroups
Mar 21 04:15:23 octo lxd.daemon[17510]: - sys_cpu_online
Mar 21 04:15:23 octo lxd.daemon[17510]: - proc_cpuinfo
Mar 21 04:15:23 octo lxd.daemon[17510]: - proc_diskstats
Mar 21 04:15:23 octo lxd.daemon[17510]: - proc_loadavg
Mar 21 04:15:23 octo lxd.daemon[17510]: - proc_meminfo
Mar 21 04:15:23 octo lxd.daemon[17510]: - proc_stat
Mar 21 04:15:23 octo lxd.daemon[17510]: - proc_swaps
Mar 21 04:15:23 octo lxd.daemon[17510]: - proc_uptime
Mar 21 04:15:23 octo lxd.daemon[17510]: - shared_pidns
Mar 21 04:15:23 octo lxd.daemon[17510]: - cpuview_daemon
Mar 21 04:15:23 octo lxd.daemon[17510]: - loadavg_daemon
Mar 21 04:15:23 octo lxd.daemon[17510]: - pidfds
Mar 21 04:15:23 octo lxd.daemon[17510]: Reloaded LXCFS
Mar 21 04:15:23 octo lxd.daemon[25258]: => Re-using existing LXCFS
Mar 21 04:15:23 octo lxd.daemon[25258]: => Starting LXD
Mar 21 04:15:23 octo lxd.daemon[25407]: t=2022-03-21T04:15:23+0800 lvl=warn msg=” - Couldn’t find the CGro>
Mar 21 04:15:23 octo lxd.daemon[25407]: t=2022-03-21T04:15:23+0800 lvl=warn msg=” - Couldn’t find the CGro>
Mar 21 04:15:24 octo lxd.daemon[25407]: t=2022-03-21T04:15:24+0800 lvl=warn msg="Dqlite: attempt 1: server>
Mar 21 04:15:24 octo lxd.daemon[25407]: t=2022-03-21T04:15:24+0800 lvl=warn msg="Dqlite: attempt 1: server>
Mar 21 04:15:24 octo lxd.daemon[25407]: t=2022-03-21T04:15:24+0800 lvl=warn msg="Dqlite: attempt 1: server>
Mar 21 04:15:24 octo lxd.daemon[25407]: t=2022-03-21T04:15:24+0800 lvl=eror msg="Failed to start the daemo>
Mar 21 04:15:24 octo lxd.daemon[25407]: t=2022-03-21T04:15:24+0800 lvl=warn msg="Could not handover member>
Mar 21 04:15:24 octo lxd.daemon[25407]: Error: Failed to initialize global database: failed to ensure sche>
Mar 21 04:15:24 octo lxd.daemon[25258]: => LXD failed to start
Mar 21 04:15:24 octo systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILU>
Mar 21 04:15:24 octo systemd[1]: snap.lxd.daemon.service: Failed with result ‘exit-code’.
Mar 21 04:15:24 octo systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 24.
Mar 21 04:15:24 octo systemd[1]: Stopped Service for snap application lxd.daemon.
Mar 21 04:15:24 octo systemd[1]: snap.lxd.daemon.service: Start request repeated too quickly.
Mar 21 04:15:24 octo systemd[1]: snap.lxd.daemon.service: Failed with result ‘exit-code’.
Mar 21 04:15:24 octo systemd[1]: Failed to start Service for snap application lxd.daemon.
What’s the current state as far as snap list lxd output on the various servers?
I have rolled back to 4.23. The output of snap list lxd in all servers is the same as follow:
Name Version Rev Tracking Publisher Notes
lxd 4.23 22652 4.23/stable canonical✓ in-cohort
The network interface can be ping from anyone of the servers.
The lxdbro0 and lxdfan0 nic disappeared in three of the servers.
Generally speaking if you have previously upgraded some of your servers to a newer version of the montly release of LXD (not the LTS) then you won’t be able to downgrade them back to the previous version (due to a DB schema changes that can occur between monthly releases).
When upgrading all members need to be on the same version. Did you manage to capture the error being show on the member that didn’t upgrade to LXD 4.24 before you downgraded them?
Thomas,
lxc list hung when I first realised that there was a version mismatch of the lxd version among the servers. I then check the lxd version one by one. I started the refresh to 4.24 in one of the servers with 4.23. It hung at the “Snap “lxd” start” prompt. I checked the the /var/snap/lxd/common/lxd/logs and saw something similar to
“Failed to initialize global database: failed to ensure schema: this node’s version is behind, please upgrade”
and a lot of lines similar to
“Dqlite: attempt #: server 192.168.1.xx:8443: no know header”
Sorry. That is all the I can recall. Since the process hung for long time and it could not be stopped, I rebooted the server and restart the refresh again. It failed.
Is there any information that I should check please?
Thanks in advance.
Terry
Yes that is normal the one that got upgraded initially was waiting for you to uprade the others to the same version. That is what I would recommend you do now.
But, the other servers hung when the lxd was refreshed to 4.24 too. When I check, only 2 out of the 5 servers were automatically upgraded to 4.24. snap changes on the failed servers reviewed that there was Error and the refresh was undone.
The VMs on the failed servers cannot be accessed. Is there anyway to start / repair the lxd please? If it is not simple to do, is it possible to retrieve the data on the inaccessible VMs please?
Thanks!
OK then we need to figure out why they can’t be brought up to LXD 4.24, that is the way out of this scenario, rather than trying to downgrade.
Please can you run snap refresh lxd --channel=latest/stable on all of your LXD servers, and then for the ones that don’t come up to LXD 4.24 provide the LXD startup error messages or snap messages?
I run snap refresh lxd --channel=latest/stable on all servers one by one. The last one refreshed successfully. However, the first 4 of them got stuck on
Start snap “lxd” (22678) services
snap list lxd showed
Name Version Rev Tracking Publisher Notes
lxd 4.24 22678 latest/stable canonical✓ in-cohort
Here is the dmesg output of one of the server.
[57162.925719] new mount options do not match the existing superblock, will be ignored
[57168.663659] new mount options do not match the existing superblock, will be ignored
[57178.420129] new mount options do not match the existing superblock, will be ignored
[57184.166305] new mount options do not match the existing superblock, will be ignored
[57192.917900] new mount options do not match the existing superblock, will be ignored
[57204.671560] new mount options do not match the existing superblock, will be ignored
[57210.417719] new mount options do not match the existing superblock, will be ignored
[57217.180935] new mount options do not match the existing superblock, will be ignored
[57224.921003] new mount options do not match the existing superblock, will be ignored
[57230.669780] new mount options do not match the existing superblock, will be ignored
[57237.411082] new mount options do not match the existing superblock, will be ignored
[57245.165732] new mount options do not match the existing superblock, will be ignored
[57249.928566] new mount options do not match the existing superblock, will be ignored
[57259.674089] new mount options do not match the existing superblock, will be ignored
[57263.432795] new mount options do not match the existing superblock, will be ignored
[57268.159381] new mount options do not match the existing superblock, will be ignored
[57273.663270] new mount options do not match the existing superblock, will be ignored
[57279.419594] new mount options do not match the existing superblock, will be ignored
[57285.162892] new mount options do not match the existing superblock, will be ignored
[57292.912976] new mount options do not match the existing superblock, will be ignored
[57299.674399] new mount options do not match the existing superblock, will be ignored
[57305.421333] new mount options do not match the existing superblock, will be ignored
[57312.163212] new mount options do not match the existing superblock, will be ignored
[57317.921987] new mount options do not match the existing superblock, will be ignored
[57323.658789] new mount options do not match the existing superblock, will be ignored
[57328.159359] new mount options do not match the existing superblock, will be ignored
[57334.031428] new mount options do not match the existing superblock, will be ignored
[57337.353463] systemd-fstab-generator[832150]: Checking was requested for “octo:/”, but it is not a device.
[57337.719251] systemd-fstab-generator[832174]: Checking was requested for “octo:/”, but it is not a device.
[57340.532989] new mount options do not match the existing superblock, will be ignored
[57341.470045] systemd-fstab-generator[832480]: Checking was requested for “octo:/”, but it is not a device.
[57341.881075] systemd-fstab-generator[832504]: Checking was requested for “octo:/”, but it is not a device.
[57342.270967] systemd-fstab-generator[832527]: Checking was requested for “octo:/”, but it is not a device.
[57342.713353] systemd-fstab-generator[832551]: Checking was requested for “octo:/”, but it is not a device.
[57343.125284] systemd-fstab-generator[832574]: Checking was requested for “octo:/”, but it is not a device.
[57343.521236] systemd-fstab-generator[832597]: Checking was requested for “octo:/”, but it is not a device.
[57347.073208] kauditd_printk_skb: 11 callbacks suppressed
[57347.073210] audit: type=1400 audit(1647863415.086:101): apparmor=“STATUS” operation=“profile_replace” info=“same as current profile, skipping” profile=“unconfined” name=“/snap/snapd/15177/usr/lib/snapd/snap-confine” pid=832632 comm=“apparmor_parser”
[57347.073218] audit: type=1400 audit(1647863415.086:102): apparmor=“STATUS” operation=“profile_replace” info=“same as current profile, skipping” profile=“unconfined” name=“/snap/snapd/15177/usr/lib/snapd/snap-confine//mount-namespace-capture-helper” pid=832632 comm=“apparmor_parser”
[57347.211072] audit: type=1400 audit(1647863415.222:103): apparmor=“STATUS” operation=“profile_replace” profile=“unconfined” name=“snap.lxd.migrate” pid=832645 comm=“apparmor_parser”
[57347.215511] audit: type=1400 audit(1647863415.226:104): apparmor=“STATUS” operation=“profile_replace” profile=“unconfined” name=“snap.lxd.lxc-to-lxd” pid=832643 comm=“apparmor_parser”
[57347.250661] audit: type=1400 audit(1647863415.262:105): apparmor=“STATUS” operation=“profile_replace” profile=“unconfined” name=“snap.lxd.activate” pid=832634 comm=“apparmor_parser”
[57347.263060] audit: type=1400 audit(1647863415.274:106): apparmor=“STATUS” operation=“profile_replace” profile=“unconfined” name=“snap.lxd.lxc” pid=832642 comm=“apparmor_parser”
[57347.266469] audit: type=1400 audit(1647863415.278:107): apparmor=“STATUS” operation=“profile_replace” profile=“unconfined” name=“snap.lxd.hook.install” pid=832640 comm=“apparmor_parser”
[57347.273361] audit: type=1400 audit(1647863415.286:108): apparmor=“STATUS” operation=“profile_replace” profile=“unconfined” name=“snap.lxd.lxd” pid=832644 comm=“apparmor_parser”
[57347.277745] audit: type=1400 audit(1647863415.290:109): apparmor=“STATUS” operation=“profile_replace” profile=“unconfined” name=“snap.lxd.hook.remove” pid=832641 comm=“apparmor_parser”
[57347.328720] audit: type=1400 audit(1647863415.342:110): apparmor=“STATUS” operation=“profile_replace” profile=“unconfined” name=“snap.lxd.benchmark” pid=832635 comm=“apparmor_parser”
[57347.839631] systemd-fstab-generator[832665]: Checking was requested for “octo:/”, but it is not a device.
[57349.292566] systemd-fstab-generator[832690]: Checking was requested for “octo:/”, but it is not a device.
[57349.688527] systemd-fstab-generator[832713]: Checking was requested for “octo:/”, but it is not a device.
[57350.115162] systemd-fstab-generator[832738]: Checking was requested for “octo:/”, but it is not a device.
[57352.426876] new mount options do not match the existing superblock, will be ignored
journalctl -u snap.lxd.daemon -n 100 showed
Mar 21 20:00:54 octo lxd.daemon[832980]: t=2022-03-21T20:00:54+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:00:58 octo lxd.daemon[832980]: t=2022-03-21T20:00:58+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
Mar 21 20:00:59 octo lxd.daemon[832980]: t=2022-03-21T20:00:59+0800 lvl=warn msg=“Failed heartbeat” err="Failed to send heartbeat request: Put "https://192.168.1.23:>
Mar 21 20:00:59 octo lxd.daemon[832980]: t=2022-03-21T20:00:59+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:01:00 octo lxd.daemon[832980]: t=2022-03-21T20:01:00+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
Mar 21 20:01:04 octo lxd.daemon[832980]: t=2022-03-21T20:01:04+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
Mar 21 20:01:05 octo lxd.daemon[832980]: t=2022-03-21T20:01:05+0800 lvl=warn msg=“Failed heartbeat” err="Failed to send heartbeat request: Put "https://192.168.1.17:>
Mar 21 20:01:05 octo lxd.daemon[832980]: t=2022-03-21T20:01:05+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:01:08 octo lxd.daemon[832980]: t=2022-03-21T20:01:08+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
Mar 21 20:01:10 octo lxd.daemon[832980]: t=2022-03-21T20:01:10+0800 lvl=warn msg=“Failed heartbeat” err="Heartbeat request failed with status: 503 Service Unavailable>
Mar 21 20:01:10 octo lxd.daemon[832980]: t=2022-03-21T20:01:10+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:01:14 octo lxd.daemon[832980]: t=2022-03-21T20:01:14+0800 lvl=warn msg=“Failed heartbeat” err="Failed to send heartbeat request: Put "https://192.168.1.17:>
Mar 21 20:01:14 octo lxd.daemon[832980]: t=2022-03-21T20:01:14+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:01:15 octo lxd.daemon[832980]: t=2022-03-21T20:01:15+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
Mar 21 20:01:17 octo lxd.daemon[832980]: t=2022-03-21T20:01:17+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
Mar 21 20:01:21 octo lxd.daemon[832980]: t=2022-03-21T20:01:21+0800 lvl=warn msg=“Failed heartbeat” err="Failed to send heartbeat request: Put "https://192.168.1.23:>
Mar 21 20:01:21 octo lxd.daemon[832980]: t=2022-03-21T20:01:21+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:01:23 octo lxd.daemon[832980]: t=2022-03-21T20:01:23+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:01:23 octo lxd.daemon[832980]: t=2022-03-21T20:01:23+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
Mar 21 20:01:25 octo lxd.daemon[832980]: t=2022-03-21T20:01:25+0800 lvl=warn msg=“Failed heartbeat” err="Failed to send heartbeat request: Put "https://192.168.1.23:>
Mar 21 20:01:25 octo lxd.daemon[832980]: t=2022-03-21T20:01:25+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:01:25 octo lxd.daemon[832980]: t=2022-03-21T20:01:25+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
Mar 21 20:01:30 octo lxd.daemon[832980]: t=2022-03-21T20:01:30+0800 lvl=warn msg=“Failed heartbeat” err="Failed to send heartbeat request: Put "https://192.168.1.17:>
Mar 21 20:01:30 octo lxd.daemon[832980]: t=2022-03-21T20:01:30+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:01:33 octo lxd.daemon[832980]: t=2022-03-21T20:01:33+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
Mar 21 20:01:34 octo lxd.daemon[832980]: t=2022-03-21T20:01:34+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
Mar 21 20:01:34 octo lxd.daemon[832980]: t=2022-03-21T20:01:34+0800 lvl=warn msg=“Failed heartbeat” err="Failed to send heartbeat request: Put "https://192.168.1.17:>
Mar 21 20:01:34 octo lxd.daemon[832980]: t=2022-03-21T20:01:34+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:01:36 octo lxd.daemon[832980]: t=2022-03-21T20:01:36+0800 lvl=warn msg=“Failed heartbeat” err="Failed to send heartbeat request: Put "https://192.168.1.23:>
Mar 21 20:01:36 octo lxd.daemon[832980]: t=2022-03-21T20:01:36+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:01:45 octo lxd.daemon[832980]: t=2022-03-21T20:01:45+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
Mar 21 20:01:47 octo lxd.daemon[832980]: t=2022-03-21T20:01:47+0800 lvl=warn msg=“Failed heartbeat” err="Failed to send heartbeat request: Put "https://192.168.1.17:>
Mar 21 20:01:47 octo lxd.daemon[832980]: t=2022-03-21T20:01:47+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:01:48 octo lxd.daemon[832980]: t=2022-03-21T20:01:48+0800 lvl=warn msg=“Failed heartbeat” err="Failed to send heartbeat request: Put "https://192.168.1.23:>
Mar 21 20:01:48 octo lxd.daemon[832980]: t=2022-03-21T20:01:48+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:01:50 octo lxd.daemon[832980]: t=2022-03-21T20:01:50+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
Mar 21 20:01:56 octo lxd.daemon[832980]: t=2022-03-21T20:01:56+0800 lvl=warn msg=“Failed heartbeat” err="Failed to send heartbeat request: Put "https://192.168.1.17:>
Mar 21 20:01:56 octo lxd.daemon[832980]: t=2022-03-21T20:01:56+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:01:57 octo lxd.daemon[832980]: t=2022-03-21T20:01:57+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
Mar 21 20:01:58 octo lxd.daemon[832980]: t=2022-03-21T20:01:58+0800 lvl=warn msg=“Failed heartbeat” err="Failed to send heartbeat request: Put "https://192.168.1.23:>
Mar 21 20:01:58 octo lxd.daemon[832980]: t=2022-03-21T20:01:58+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:02:00 octo lxd.daemon[832980]: t=2022-03-21T20:02:00+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
Mar 21 20:02:05 octo lxd.daemon[832980]: t=2022-03-21T20:02:05+0800 lvl=warn msg=“Failed heartbeat” err="Failed to send heartbeat request: Put "https://192.168.1.23:>
Mar 21 20:02:05 octo lxd.daemon[832980]: t=2022-03-21T20:02:05+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:02:06 octo lxd.daemon[832980]: t=2022-03-21T20:02:06+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
Mar 21 20:02:09 octo lxd.daemon[832980]: t=2022-03-21T20:02:09+0800 lvl=warn msg=“Failed heartbeat” err="Failed to send heartbeat request: Put "https://192.168.1.17:>
Mar 21 20:02:09 octo lxd.daemon[832980]: t=2022-03-21T20:02:09+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:02:10 octo lxd.daemon[832980]: t=2022-03-21T20:02:10+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
Mar 21 20:02:16 octo lxd.daemon[832980]: t=2022-03-21T20:02:16+0800 lvl=warn msg=“Failed heartbeat” err="Failed to send heartbeat request: Put "https://192.168.1.17:>
Mar 21 20:02:16 octo lxd.daemon[832980]: t=2022-03-21T20:02:16+0800 lvl=warn msg=“Failed to create warning” err=“Local member name not available”
Mar 21 20:02:18 octo lxd.daemon[832980]: t=2022-03-21T20:02:18+0800 lvl=warn msg=“Failed to resolve warning” err=“Local member name not available”
These are just warnings and won’t prevent LXD starting, can you expand the amount of log lines you’re looking at and see if there is another reason?
That is normal they are waiting for all members to be on same version, did you cancel this process?
There are some error messages related to
Failed to initialize global database: failed to ensure schema: this node’s version is behind, please upgrade
Here is the output of journalctl -u snap.lxd.daemon -n 300
– Logs begin at Sat 2021-01-02 19:04:58 HKT, end at Mon 2022-03-21 20:31:20 HKT. –
Mar 21 20:06:48 piaget lxd.daemon[2239]: Running destructor lxcfs_exit
Mar 21 20:06:48 piaget lxd.daemon[2239]: Running constructor lxcfs_init to reload liblxcfs
Mar 21 20:06:48 piaget lxd.daemon[2239]: mount namespace: 5
Mar 21 20:06:48 piaget lxd.daemon[2239]: hierarchies:
Mar 21 20:06:48 piaget lxd.daemon[2239]: 0: fd: 6:
Mar 21 20:06:48 piaget lxd.daemon[2239]: 1: fd: 7: name=systemd
Mar 21 20:06:48 piaget lxd.daemon[2239]: 2: fd: 8: net_cls,net_prio
Mar 21 20:06:48 piaget lxd.daemon[2239]: 3: fd: 9: memory
gccadmin@piaget:~$ sudo journalctl -u snap.lxd.daemon -n 300
– Logs begin at Sat 2021-01-02 19:04:58 HKT, end at Mon 2022-03-21 20:34:40 HKT. –
Mar 21 20:06:48 piaget lxd.daemon[2239]: hierarchies:
Mar 21 20:06:48 piaget lxd.daemon[2239]: 0: fd: 6:
Mar 21 20:06:48 piaget lxd.daemon[2239]: 1: fd: 7: name=systemd
Mar 21 20:06:48 piaget lxd.daemon[2239]: 2: fd: 8: net_cls,net_prio
Mar 21 20:06:48 piaget lxd.daemon[2239]: 3: fd: 9: memory
Mar 21 20:06:48 piaget lxd.daemon[2239]: 4: fd: 10: freezer
Mar 21 20:06:48 piaget lxd.daemon[2239]: 5: fd: 11: perf_event
Mar 21 20:06:48 piaget lxd.daemon[2239]: 6: fd: 12: rdma
Mar 21 20:06:48 piaget lxd.daemon[2239]: 7: fd: 13: devices
Mar 21 20:06:48 piaget lxd.daemon[2239]: 8: fd: 14: cpu,cpuacct
Mar 21 20:06:48 piaget lxd.daemon[2239]: 9: fd: 15: cpuset
Mar 21 20:06:48 piaget lxd.daemon[2239]: 10: fd: 16: pids
Mar 21 20:06:48 piaget lxd.daemon[2239]: 11: fd: 17: hugetlb
Mar 21 20:06:48 piaget lxd.daemon[2239]: 12: fd: 19: blkio
Mar 21 20:06:48 piaget lxd.daemon[2239]: Kernel supports pidfds
Mar 21 20:06:48 piaget lxd.daemon[2239]: Kernel does not support swap accounting
Mar 21 20:06:48 piaget lxd.daemon[2239]: api_extensions:
Mar 21 20:06:48 piaget lxd.daemon[2239]: - cgroups
Mar 21 20:06:48 piaget lxd.daemon[2239]: - sys_cpu_online
Mar 21 20:06:48 piaget lxd.daemon[2239]: - proc_cpuinfo
Mar 21 20:06:48 piaget lxd.daemon[2239]: - proc_diskstats
Mar 21 20:06:48 piaget lxd.daemon[2239]: - proc_loadavg
Mar 21 20:06:48 piaget lxd.daemon[2239]: - proc_meminfo
Mar 21 20:06:48 piaget lxd.daemon[2239]: - proc_stat
Mar 21 20:06:48 piaget lxd.daemon[2239]: - proc_swaps
Mar 21 20:06:48 piaget lxd.daemon[2239]: - proc_uptime
Mar 21 20:06:48 piaget lxd.daemon[2239]: - shared_pidns
Mar 21 20:06:48 piaget lxd.daemon[2239]: - cpuview_daemon
Mar 21 20:06:48 piaget lxd.daemon[2239]: - loadavg_daemon
Mar 21 20:06:48 piaget lxd.daemon[2239]: - pidfds
Mar 21 20:06:48 piaget lxd.daemon[2239]: Reloaded LXCFS
Mar 21 20:06:48 piaget lxd.daemon[1582759]: => Re-using existing LXCFS
Mar 21 20:06:48 piaget lxd.daemon[1582759]: ==> Cleaning up existing LXCFS namespace
Mar 21 20:06:49 piaget lxd.daemon[1582759]: => Starting LXD
Mar 21 20:06:49 piaget lxd.daemon[1583311]: t=2022-03-21T20:06:49+0800 lvl=warn msg=" - Couldn’t find the CGroup blkio.weight, disk priority will be ignored"
Mar 21 20:06:49 piaget lxd.daemon[1583311]: t=2022-03-21T20:06:49+0800 lvl=warn msg=" - Couldn’t find the CGroup memory swap accounting, swap limits will be ignored"
Mar 21 20:06:49 piaget lxd.daemon[1583311]: t=2022-03-21T20:06:49+0800 lvl=eror msg=“Failed to start the daemon” err=“Failed to initialize global database: failed to ensure schema: this node’s version is behind, please upgrade”
Mar 21 20:06:50 piaget lxd.daemon[1583311]: t=2022-03-21T20:06:50+0800 lvl=warn msg=“Could not handover member’s responsibilities” err=“LXD daemon setup in progress”
Mar 21 20:06:51 piaget lxd.daemon[1583311]: Error: Failed to initialize global database: failed to ensure schema: this node’s version is behind, please upgrade
Mar 21 20:06:52 piaget lxd.daemon[1582759]: => LXD failed to start
Mar 21 20:06:52 piaget lxd.daemon[2239]: Closed liblxcfs.so
Mar 21 20:06:52 piaget lxd.daemon[2239]: Running destructor lxcfs_exit
Mar 21 20:06:52 piaget lxd.daemon[2239]: Running constructor lxcfs_init to reload liblxcfs
Mar 21 20:06:52 piaget systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Mar 21 20:06:52 piaget systemd[1]: snap.lxd.daemon.service: Failed with result ‘exit-code’.
Mar 21 20:06:52 piaget lxd.daemon[2239]: mount namespace: 5
Mar 21 20:06:52 piaget lxd.daemon[2239]: hierarchies:
Mar 21 20:06:52 piaget lxd.daemon[2239]: 0: fd: 6:
Mar 21 20:06:52 piaget lxd.daemon[2239]: 1: fd: 7: name=systemd
Mar 21 20:06:52 piaget lxd.daemon[2239]: 2: fd: 8: net_cls,net_prio
Mar 21 20:06:52 piaget lxd.daemon[2239]: 3: fd: 9: memory
Mar 21 20:06:52 piaget lxd.daemon[2239]: 4: fd: 10: freezer
Mar 21 20:06:52 piaget lxd.daemon[2239]: 5: fd: 11: perf_event
Mar 21 20:06:52 piaget lxd.daemon[2239]: 6: fd: 12: rdma
Mar 21 20:06:52 piaget lxd.daemon[2239]: 7: fd: 13: devices
Mar 21 20:06:52 piaget lxd.daemon[2239]: 8: fd: 14: cpu,cpuacct
Mar 21 20:06:52 piaget lxd.daemon[2239]: 9: fd: 15: cpuset
Mar 21 20:06:52 piaget lxd.daemon[2239]: 10: fd: 16: pids
Mar 21 20:06:52 piaget lxd.daemon[2239]: 11: fd: 17: hugetlb
Mar 21 20:06:52 piaget lxd.daemon[2239]: 12: fd: 19: blkio
Mar 21 20:06:52 piaget lxd.daemon[2239]: Kernel supports pidfds
Mar 21 20:06:52 piaget lxd.daemon[2239]: Kernel does not support swap accounting
Mar 21 20:06:52 piaget lxd.daemon[2239]: api_extensions:
Mar 21 20:06:52 piaget lxd.daemon[2239]: - cgroups
Mar 21 20:06:52 piaget lxd.daemon[2239]: - sys_cpu_online
Mar 21 20:06:52 piaget lxd.daemon[2239]: - proc_cpuinfo
Mar 21 20:06:52 piaget lxd.daemon[2239]: - proc_diskstats
Mar 21 20:06:52 piaget lxd.daemon[2239]: - proc_loadavg
Mar 21 20:06:52 piaget lxd.daemon[2239]: - proc_meminfo
Mar 21 20:06:52 piaget lxd.daemon[2239]: - proc_stat
Mar 21 20:06:52 piaget lxd.daemon[2239]: - proc_swaps
Mar 21 20:06:52 piaget lxd.daemon[2239]: - proc_uptime
Mar 21 20:06:52 piaget lxd.daemon[2239]: - shared_pidns
Mar 21 20:06:52 piaget lxd.daemon[2239]: - cpuview_daemon
Mar 21 20:06:52 piaget lxd.daemon[2239]: - loadavg_daemon
Mar 21 20:06:52 piaget lxd.daemon[2239]: - pidfds
Mar 21 20:06:52 piaget lxd.daemon[2239]: Reloaded LXCFS
Mar 21 20:06:52 piaget systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 5203.
Mar 21 20:06:52 piaget systemd[1]: Stopped Service for snap application lxd.daemon.
Mar 21 20:06:52 piaget systemd[1]: Started Service for snap application lxd.daemon.
Mar 21 20:06:52 piaget lxd.daemon[1583372]: => Preparing the system (22652)
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Loading snap configuration
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Setting up mntns symlink (mnt:[4026533095])
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Setting up kmod wrapper
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Preparing /boot
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Preparing a clean copy of /run
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Preparing /run/bin
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Preparing a clean copy of /etc
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Preparing a clean copy of /usr/share/misc
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Setting up ceph configuration
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Setting up LVM configuration
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Setting up OVN configuration
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Rotating logs
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Setting up ZFS (0.8)
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Escaping the systemd cgroups
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ====> Detected cgroup V1
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Escaping the systemd process resource limits
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Disabling shiftfs on this kernel (auto)
Mar 21 20:06:52 piaget lxd.daemon[1583372]: => Re-using existing LXCFS
Mar 21 20:06:52 piaget lxd.daemon[1583372]: ==> Cleaning up existing LXCFS namespace
Mar 21 20:06:53 piaget lxd.daemon[1583372]: => Starting LXD
Mar 21 20:06:53 piaget lxd.daemon[1583926]: t=2022-03-21T20:06:53+0800 lvl=warn msg=" - Couldn’t find the CGroup blkio.weight, disk priority will be ignored"
Mar 21 20:06:53 piaget lxd.daemon[1583926]: t=2022-03-21T20:06:53+0800 lvl=warn msg=" - Couldn’t find the CGroup memory swap accounting, swap limits will be ignored"
Mar 21 20:06:53 piaget lxd.daemon[2239]: Closed liblxcfs.so
Mar 21 20:06:53 piaget lxd.daemon[2239]: Running destructor lxcfs_exit
Mar 21 20:06:53 piaget lxd.daemon[2239]: Running constructor lxcfs_init to reload liblxcfs
Mar 21 20:06:53 piaget lxd.daemon[2239]: mount namespace: 5
Mar 21 20:06:53 piaget lxd.daemon[2239]: hierarchies:
Mar 21 20:06:53 piaget lxd.daemon[2239]: 0: fd: 6:
Mar 21 20:06:53 piaget lxd.daemon[2239]: 1: fd: 7: name=systemd
Mar 21 20:06:53 piaget lxd.daemon[2239]: 2: fd: 8: net_cls,net_prio
Mar 21 20:06:53 piaget lxd.daemon[2239]: 3: fd: 9: memory
Mar 21 20:06:53 piaget lxd.daemon[2239]: 4: fd: 10: freezer
Mar 21 20:06:53 piaget lxd.daemon[2239]: 5: fd: 11: perf_event
Mar 21 20:06:53 piaget lxd.daemon[2239]: 6: fd: 12: rdma
Mar 21 20:06:53 piaget lxd.daemon[2239]: 7: fd: 13: devices
Mar 21 20:06:53 piaget lxd.daemon[2239]: 8: fd: 14: cpu,cpuacct
Mar 21 20:06:53 piaget lxd.daemon[2239]: 9: fd: 15: cpuset
Mar 21 20:06:53 piaget lxd.daemon[2239]: 10: fd: 16: pids
Mar 21 20:06:53 piaget lxd.daemon[2239]: 11: fd: 17: hugetlb
Mar 21 20:06:53 piaget lxd.daemon[2239]: 12: fd: 19: blkio
Mar 21 20:06:53 piaget lxd.daemon[2239]: Kernel supports pidfds
Mar 21 20:06:53 piaget lxd.daemon[2239]: Kernel does not support swap accounting
Mar 21 20:06:53 piaget lxd.daemon[2239]: api_extensions:
Mar 21 20:06:53 piaget lxd.daemon[2239]: - cgroups
Mar 21 20:06:53 piaget lxd.daemon[2239]: - sys_cpu_online
Mar 21 20:06:53 piaget lxd.daemon[2239]: - proc_cpuinfo
Mar 21 20:06:53 piaget lxd.daemon[2239]: - proc_diskstats
Mar 21 20:06:53 piaget lxd.daemon[2239]: - proc_loadavg
Mar 21 20:06:53 piaget lxd.daemon[2239]: - proc_meminfo
Mar 21 20:06:53 piaget lxd.daemon[2239]: - proc_stat
Mar 21 20:06:53 piaget lxd.daemon[2239]: - proc_swaps
Mar 21 20:06:53 piaget lxd.daemon[2239]: - proc_uptime
Mar 21 20:06:53 piaget lxd.daemon[2239]: - shared_pidns
Mar 21 20:06:53 piaget lxd.daemon[2239]: - cpuview_daemon
Mar 21 20:06:53 piaget lxd.daemon[2239]: - loadavg_daemon
Mar 21 20:06:53 piaget lxd.daemon[2239]: - pidfds
Mar 21 20:06:53 piaget lxd.daemon[2239]: Reloaded LXCFS
Mar 21 20:06:53 piaget lxd.daemon[1583926]: t=2022-03-21T20:06:53+0800 lvl=eror msg=“Failed to start the daemon” err=“Failed to initialize global database: failed to ensure schema: this node’s version is behind, please upgrade”
Mar 21 20:06:54 piaget lxd.daemon[1583926]: t=2022-03-21T20:06:54+0800 lvl=warn msg=“Could not handover member’s responsibilities” err=“LXD daemon setup in progress”
Mar 21 20:06:55 piaget lxd.daemon[1583926]: Error: Failed to initialize global database: failed to ensure schema: this node’s version is behind, please upgrade
Mar 21 20:06:55 piaget lxd.daemon[1583372]: => LXD failed to start
Mar 21 20:06:55 piaget systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Mar 21 20:06:55 piaget systemd[1]: snap.lxd.daemon.service: Failed with result ‘exit-code’.
Mar 21 20:06:55 piaget systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 5204.
Mar 21 20:06:55 piaget systemd[1]: Stopped Service for snap application lxd.daemon.
Mar 21 20:06:55 piaget systemd[1]: Started Service for snap application lxd.daemon.
Mar 21 20:06:55 piaget lxd.daemon[1584014]: => Preparing the system (22652)
Mar 21 20:06:55 piaget lxd.daemon[1584014]: ==> Loading snap configuration
Mar 21 20:06:55 piaget lxd.daemon[1584014]: ==> Setting up mntns symlink (mnt:[4026533095])
Mar 21 20:06:55 piaget lxd.daemon[1584014]: ==> Setting up kmod wrapper
Mar 21 20:06:55 piaget lxd.daemon[1584014]: ==> Preparing /boot
Mar 21 20:06:55 piaget lxd.daemon[1584014]: ==> Preparing a clean copy of /run
Mar 21 20:06:56 piaget lxd.daemon[1584014]: ==> Preparing /run/bin
Mar 21 20:06:56 piaget lxd.daemon[1584014]: ==> Preparing a clean copy of /etc
Mar 21 20:06:56 piaget lxd.daemon[1584014]: ==> Preparing a clean copy of /usr/share/misc
Mar 21 20:06:56 piaget lxd.daemon[1584014]: ==> Setting up ceph configuration
Mar 21 20:06:56 piaget lxd.daemon[1584014]: ==> Setting up LVM configuration
Mar 21 20:06:56 piaget lxd.daemon[1584014]: ==> Setting up OVN configuration
Mar 21 20:06:56 piaget lxd.daemon[1584014]: ==> Rotating logs
Mar 21 20:06:56 piaget lxd.daemon[1584014]: ==> Setting up ZFS (0.8)
Mar 21 20:06:56 piaget lxd.daemon[1584014]: ==> Escaping the systemd cgroups
Mar 21 20:06:56 piaget lxd.daemon[1584014]: ====> Detected cgroup V1
Mar 21 20:06:56 piaget lxd.daemon[1584014]: ==> Escaping the systemd process resource limits
Mar 21 20:06:56 piaget lxd.daemon[1584014]: ==> Disabling shiftfs on this kernel (auto)
Mar 21 20:06:56 piaget lxd.daemon[1584014]: => Re-using existing LXCFS
Mar 21 20:06:56 piaget lxd.daemon[1584014]: ==> Cleaning up existing LXCFS namespace
Mar 21 20:06:56 piaget systemd[1]: Stopping Service for snap application lxd.daemon…
Mar 21 20:06:56 piaget lxd.daemon[1584014]: => Starting LXD
Mar 21 20:06:56 piaget lxd.daemon[1584562]: => Stop reason is: snap refresh
Mar 21 20:06:56 piaget lxd.daemon[1584562]: => Stopping LXD
Mar 21 20:06:57 piaget lxd.daemon[1584014]: => LXD failed to start
Mar 21 20:06:57 piaget systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Mar 21 20:06:57 piaget lxd.daemon[1584562]: ==> Stopped LXD
Mar 21 20:06:57 piaget systemd[1]: snap.lxd.daemon.service: Failed with result ‘exit-code’.
Mar 21 20:06:57 piaget systemd[1]: Stopped Service for snap application lxd.daemon.
Mar 21 20:07:09 piaget systemd[1]: Started Service for snap application lxd.daemon.
Mar 21 20:07:10 piaget lxd.daemon[1585213]: => Preparing the system (22678)
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Loading snap configuration
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Setting up mntns symlink (mnt:[4026533095])
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Setting up kmod wrapper
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Preparing /boot
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Preparing a clean copy of /run
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Preparing /run/bin
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Preparing a clean copy of /etc
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Preparing a clean copy of /usr/share/misc
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Setting up ceph configuration
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Setting up LVM configuration
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Setting up OVN configuration
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Rotating logs
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Setting up ZFS (0.8)
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Escaping the systemd cgroups
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ====> Detected cgroup V1
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Escaping the systemd process resource limits
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Disabling shiftfs on this kernel (auto)
Mar 21 20:07:10 piaget lxd.daemon[2239]: Closed liblxcfs.so
Mar 21 20:07:10 piaget lxd.daemon[2239]: Running destructor lxcfs_exit
Mar 21 20:07:10 piaget lxd.daemon[2239]: Running constructor lxcfs_init to reload liblxcfs
Mar 21 20:07:10 piaget lxd.daemon[2239]: mount namespace: 5
Mar 21 20:07:10 piaget lxd.daemon[2239]: hierarchies:
Mar 21 20:07:10 piaget lxd.daemon[2239]: 0: fd: 6:
Mar 21 20:07:10 piaget lxd.daemon[2239]: 1: fd: 7: name=systemd
Mar 21 20:07:10 piaget lxd.daemon[2239]: 2: fd: 8: net_cls,net_prio
Mar 21 20:07:10 piaget lxd.daemon[2239]: 3: fd: 9: memory
Mar 21 20:07:10 piaget lxd.daemon[2239]: 4: fd: 10: freezer
Mar 21 20:07:10 piaget lxd.daemon[2239]: 5: fd: 11: perf_event
Mar 21 20:07:10 piaget lxd.daemon[2239]: 6: fd: 12: rdma
Mar 21 20:07:10 piaget lxd.daemon[2239]: 7: fd: 13: devices
Mar 21 20:07:10 piaget lxd.daemon[2239]: 8: fd: 14: cpu,cpuacct
Mar 21 20:07:10 piaget lxd.daemon[2239]: 9: fd: 15: cpuset
Mar 21 20:07:10 piaget lxd.daemon[2239]: 10: fd: 16: pids
Mar 21 20:07:10 piaget lxd.daemon[2239]: 11: fd: 17: hugetlb
Mar 21 20:07:10 piaget lxd.daemon[2239]: 12: fd: 19: blkio
Mar 21 20:07:10 piaget lxd.daemon[2239]: Kernel supports pidfds
Mar 21 20:07:10 piaget lxd.daemon[2239]: Kernel does not support swap accounting
Mar 21 20:07:10 piaget lxd.daemon[2239]: api_extensions:
Mar 21 20:07:10 piaget lxd.daemon[2239]: - cgroups
Mar 21 20:07:10 piaget lxd.daemon[2239]: - sys_cpu_online
Mar 21 20:07:10 piaget lxd.daemon[2239]: - proc_cpuinfo
Mar 21 20:07:10 piaget lxd.daemon[2239]: - proc_diskstats
Mar 21 20:07:10 piaget lxd.daemon[2239]: - proc_loadavg
Mar 21 20:07:10 piaget lxd.daemon[2239]: - proc_meminfo
Mar 21 20:07:10 piaget lxd.daemon[2239]: - proc_stat
Mar 21 20:07:10 piaget lxd.daemon[2239]: - proc_swaps
Mar 21 20:07:10 piaget lxd.daemon[2239]: - proc_uptime
Mar 21 20:07:10 piaget lxd.daemon[2239]: - proc_slabinfo
Mar 21 20:07:10 piaget lxd.daemon[2239]: - shared_pidns
Mar 21 20:07:10 piaget lxd.daemon[2239]: - cpuview_daemon
Mar 21 20:07:10 piaget lxd.daemon[2239]: - loadavg_daemon
Mar 21 20:07:10 piaget lxd.daemon[2239]: - pidfds
Mar 21 20:07:10 piaget lxd.daemon[2239]: Reloaded LXCFS
Mar 21 20:07:10 piaget lxd.daemon[1585213]: => Re-using existing LXCFS
Mar 21 20:07:10 piaget lxd.daemon[1585213]: ==> Cleaning up existing LXCFS namespace
Mar 21 20:07:10 piaget lxd.daemon[1585213]: => Starting LXD
Mar 21 20:07:10 piaget lxd.daemon[1585761]: t=2022-03-21T20:07:10+0800 lvl=warn msg=" - Couldn’t find the CGroup blkio.weight, disk priority will be ignored"
Mar 21 20:07:10 piaget lxd.daemon[1585761]: t=2022-03-21T20:07:10+0800 lvl=warn msg=" - Couldn’t find the CGroup memory swap accounting, swap limits will be ignored"
Mar 21 20:07:11 piaget lxd.daemon[1585761]: t=2022-03-21T20:07:11+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:07:27 piaget lxd.daemon[2239]: Closed liblxcfs.so
Mar 21 20:07:27 piaget lxd.daemon[2239]: Running destructor lxcfs_exit
Mar 21 20:07:27 piaget lxd.daemon[2239]: Running constructor lxcfs_init to reload liblxcfs
Mar 21 20:07:27 piaget lxd.daemon[2239]: mount namespace: 5
Mar 21 20:07:27 piaget lxd.daemon[2239]: hierarchies:
Mar 21 20:07:27 piaget lxd.daemon[2239]: 0: fd: 6:
Mar 21 20:07:27 piaget lxd.daemon[2239]: 1: fd: 7: name=systemd
Mar 21 20:07:27 piaget lxd.daemon[2239]: 2: fd: 8: net_cls,net_prio
Mar 21 20:07:27 piaget lxd.daemon[2239]: 3: fd: 9: memory
Mar 21 20:07:27 piaget lxd.daemon[2239]: 4: fd: 10: freezer
Mar 21 20:07:27 piaget lxd.daemon[2239]: 5: fd: 11: perf_event
Mar 21 20:07:27 piaget lxd.daemon[2239]: 6: fd: 12: rdma
Mar 21 20:07:27 piaget lxd.daemon[2239]: 7: fd: 13: devices
Mar 21 20:07:27 piaget lxd.daemon[2239]: 8: fd: 14: cpu,cpuacct
Mar 21 20:07:27 piaget lxd.daemon[2239]: 9: fd: 15: cpuset
Mar 21 20:07:27 piaget lxd.daemon[2239]: 10: fd: 16: pids
Mar 21 20:07:27 piaget lxd.daemon[2239]: 11: fd: 17: hugetlb
Mar 21 20:07:27 piaget lxd.daemon[2239]: 12: fd: 19: blkio
Mar 21 20:07:27 piaget lxd.daemon[2239]: Kernel supports pidfds
Mar 21 20:07:27 piaget lxd.daemon[2239]: Kernel does not support swap accounting
Mar 21 20:07:27 piaget lxd.daemon[2239]: api_extensions:
Mar 21 20:07:27 piaget lxd.daemon[2239]: - cgroups
Mar 21 20:07:27 piaget lxd.daemon[2239]: - sys_cpu_online
Mar 21 20:07:27 piaget lxd.daemon[2239]: - proc_cpuinfo
Mar 21 20:07:27 piaget lxd.daemon[2239]: - proc_diskstats
Mar 21 20:07:27 piaget lxd.daemon[2239]: - proc_loadavg
Mar 21 20:07:27 piaget lxd.daemon[2239]: - proc_meminfo
Mar 21 20:07:27 piaget lxd.daemon[2239]: - proc_stat
Mar 21 20:07:27 piaget lxd.daemon[2239]: - proc_swaps
Mar 21 20:07:27 piaget lxd.daemon[2239]: - proc_uptime
Mar 21 20:07:27 piaget lxd.daemon[2239]: - proc_slabinfo
Mar 21 20:07:27 piaget lxd.daemon[2239]: - shared_pidns
Mar 21 20:07:27 piaget lxd.daemon[2239]: - cpuview_daemon
Mar 21 20:07:27 piaget lxd.daemon[2239]: - loadavg_daemon
Mar 21 20:07:27 piaget lxd.daemon[2239]: - pidfds
Mar 21 20:07:27 piaget lxd.daemon[2239]: Reloaded LXCFS
Mar 21 20:08:11 piaget lxd.daemon[1585761]: t=2022-03-21T20:08:11+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:09:11 piaget lxd.daemon[1585761]: t=2022-03-21T20:09:11+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:10:11 piaget lxd.daemon[1585761]: t=2022-03-21T20:10:11+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:11:11 piaget lxd.daemon[1585761]: t=2022-03-21T20:11:11+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:12:11 piaget lxd.daemon[1585761]: t=2022-03-21T20:12:11+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:13:11 piaget lxd.daemon[1585761]: t=2022-03-21T20:13:11+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:14:12 piaget lxd.daemon[1585761]: t=2022-03-21T20:14:12+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:15:12 piaget lxd.daemon[1585761]: t=2022-03-21T20:15:12+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:16:12 piaget lxd.daemon[1585761]: t=2022-03-21T20:16:12+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:17:12 piaget lxd.daemon[1585761]: t=2022-03-21T20:17:12+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:18:12 piaget lxd.daemon[1585761]: t=2022-03-21T20:18:12+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:19:12 piaget lxd.daemon[1585761]: t=2022-03-21T20:19:12+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:20:12 piaget lxd.daemon[1585761]: t=2022-03-21T20:20:12+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:21:12 piaget lxd.daemon[1585761]: t=2022-03-21T20:21:12+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:22:12 piaget lxd.daemon[1585761]: t=2022-03-21T20:22:12+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:23:12 piaget lxd.daemon[1585761]: t=2022-03-21T20:23:12+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:24:12 piaget lxd.daemon[1585761]: t=2022-03-21T20:24:12+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:25:12 piaget lxd.daemon[1585761]: t=2022-03-21T20:25:12+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:26:13 piaget lxd.daemon[1585761]: t=2022-03-21T20:26:13+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:27:13 piaget lxd.daemon[1585761]: t=2022-03-21T20:27:13+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:28:13 piaget lxd.daemon[1585761]: t=2022-03-21T20:28:13+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:29:13 piaget lxd.daemon[1585761]: t=2022-03-21T20:29:13+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:30:13 piaget lxd.daemon[1585761]: t=2022-03-21T20:30:13+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:31:13 piaget lxd.daemon[1585761]: t=2022-03-21T20:31:13+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:32:13 piaget lxd.daemon[1585761]: t=2022-03-21T20:32:13+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:33:13 piaget lxd.daemon[1585761]: t=2022-03-21T20:33:13+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Mar 21 20:34:13 piaget lxd.daemon[1585761]: t=2022-03-21T20:34:13+0800 lvl=warn msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
lines 266-301/301 (END)
Yes, I did stop the hung process of
“Start snap “lxd” (22678) services”
by rebooting the servers but not this time.
OK so it looks like its still waiting for the other members to be upgraded to LXD 4.24.
Please can you show the output of snap info lxd on each server.
the output of snap info lxd in all servers showed the same result as below:
name: lxd
summary: LXD - container and VM manager
publisher: Canonical✓
store-url: https://snapcraft.io/lxd
contact: Issues · lxc/incus · GitHub
license: unset
description: |
LXD is a system container and virtual machine manager.It offers a simple CLI and REST API to manage local or remote instances,
uses an image based workflow and support for a variety of advanced features.Images are available for all Ubuntu releases and architectures as well
as for a wide number of other Linux distributions. Existing
integrations with many deployment and operation tools, makes it work
just like a public cloud, except everything is under your control.LXD containers are lightweight, secure by default and a great
alternative to virtual machines when running Linux on Linux.LXD virtual machines are modern and secure, using UEFI and secure-boot
by default and a great choice when a different kernel or operating
system is needed.With clustering, up to 50 LXD servers can be easily joined and managed
together with the same tools and APIs and without needing any external
dependencies.Supported configuration options for the snap (snap set lxd [=…]):
- ceph.builtin: Use snap-specific Ceph configuration [default=false] - ceph.external: Use the system's ceph tools (ignores ceph.builtin) [default=false] - criu.enable: Enable experimental live-migration support [default=false] - daemon.debug: Increase logging to debug level [default=false] - daemon.group: Set group of users that have full control over LXD [default=lxd] - daemon.user.group: Set group of users that have restricted LXD access [default=lxd] - daemon.preseed: Pass a YAML configuration to `lxd init` on initial start - daemon.syslog: Send LXD log events to syslog [default=false] - lvm.external: Use the system's LVM tools [default=false] - lxcfs.pidfd: Start per-container process tracking [default=false] - lxcfs.loadavg: Start tracking per-container load average [default=false] - lxcfs.cfs: Consider CPU shares for CPU usage [default=false] - openvswitch.builtin: Run a snap-specific OVS daemon [default=false] - ovn.builtin: Use snap-specific OVN configuration [default=false] - shiftfs.enable: Enable shiftfs support [default=auto]For system-wide configuration of the CLI, place your configuration in
/var/snap/lxd/common/global-conf/ (config.yml and servercerts)
commands:
- lxd.benchmark
- lxd.buginfo
- lxd.check-kernel
- lxd.lxc
- lxd.lxc-to-lxd
- lxd
- lxd.migrate
services:
lxd.activate: oneshot, enabled, inactive
lxd.daemon: simple, enabled, active
lxd.user-daemon: simple, enabled, inactivesnap-id: J60k4JY0HppjwOjW8dZdYc8obXKxujRu
tracking: latest/stable
refresh-date: today at 20:07 HKT
channels:
latest/stable: 4.24 2022-03-19 (22678) 82MB -
latest/candidate: 4.24 2022-03-18 (22678) 82MB -
latest/beta: 4.23 2022-03-12 (22652) 82MB -
latest/edge: git-7d6bf1b 2022-03-21 (22701) 82MB -
4.24/stable: 4.24 2022-03-19 (22678) 82MB -
4.24/candidate: 4.24 2022-03-18 (22678) 82MB -
4.24/beta: ↑
4.24/edge: ↑
4.23/stable: 4.23 2022-03-13 (22652) 82MB -
4.23/candidate: 4.23 2022-03-10 (22633) 82MB -
4.23/beta: ↑
4.23/edge: ↑
4.22/stable: 4.22 2022-02-12 (22407) 79MB -
4.22/candidate: 4.22 2022-02-11 (22407) 79MB -
4.22/beta: ↑
4.22/edge: ↑
4.0/stable: 4.0.9 2022-02-25 (22526) 71MB -
4.0/candidate: 4.0.9 2022-02-24 (22541) 71MB -
4.0/beta: ↑
4.0/edge: git-407205d 2022-03-19 (22696) 71MB -
3.0/stable: 3.0.4 2019-10-10 (11348) 55MB -
3.0/candidate: 3.0.4 2019-10-10 (11348) 55MB -
3.0/beta: ↑
3.0/edge: git-81b81b9 2019-10-10 (11362) 55MB -
2.0/stable: 2.0.12 2020-08-18 (16879) 38MB -
2.0/candidate: 2.0.12 2021-03-22 (19859) 39MB -
2.0/beta: ↑
2.0/edge: git-82c7d62 2021-03-22 (19857) 39MB -
installed: 4.24 (22678) 82MB in-cohort
The refresh-dates are different. Do you want to see all of them?
I’m most interested in this like
Finally, all refreshes were completed:
snap refresh lxd --channel=latest/stable
lxd 4.24 from Canonical✓ refreshed
after almost 3 hours!!!
Thank you so much!
Time for me to do some clean up and back up!
Thanks mate!
Hi Thomas,
I restarted the server that was the 1st refreshed one. All lxd nic were gone after the reboot. Could you please tell me how to get the lxd started on this server?
When I run lxc list, I’ve got this:
Error: Get “http://unix.socket/1.0”: dial unix /var/snap/lxd/common/lxd/unix.socket: connect: permission denied
Terry