Hi, I have a lxd cluster of 3 nodes, however, one node failed to upgrade:
root@qa-physical-5-4:~# snap refresh lxd
error: cannot perform the following tasks:
- Copy snap "lxd" data (cannot find installed snap "lxd" at revision 24323: missing file /snap/lxd/24323/meta/snap.yaml)
root@qa-physical-5-4:~# ll /snap/lxd
total 8
drwxr-xr-x 4 root root 4096 Jan 29 02:17 ./
drwxr-xr-x 6 root root 4096 Dec 6 03:56 ../
drwxr-xr-x 20 root root 276 Dec 16 18:07 24164/
drwxr-xr-x 20 root root 276 Jan 5 14:16 24175/
lrwxrwxrwx 1 root root 5 Jan 29 02:17 current -> 24175/
root@qa-physical-5-4:~# snap list --all
Name Version Rev Tracking Publisher Notes
core20 20221123 1738 latest/stable canonical✓ base,disabled
core20 20221212 1778 latest/stable canonical✓ base
lxd 5.9-76c110d 24164 latest/stable/… canonical✓ -
lxd 5.9-9879096 24175 latest/stable/… canonical✓ disabled
snapd 2.57.6 17883 latest/stable canonical✓ snapd,disabled
snapd 2.58 17950 latest/stable canonical✓ snapd
And the lxd cluster can’t be started:
2023-01-29T02:16:55Z lxd.daemon[2023141]: time="2023-01-29T02:16:55Z" level=error msg="Failed to start the daemon" err="Failed to initialize global database: failed to ensure schema: this node's version is behind, please upgrade"
I have already restart the snap service, it does not work. Anyone can help me ?
root@qa-physical-5-4:~# systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● snap-lxd-23889.mount not-found failed failed snap-lxd-23889.mount
● snap-lxd-23983.mount not-found failed failed snap-lxd-23983.mount
● snap-snapd-17576.mount not-found failed failed snap-snapd-17576.mount
● fwupd-refresh.service loaded failed failed Refresh fwupd metadata and update motd
● systemd-networkd-wait-online.service loaded failed failed Wait for Network to be Configured
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
5 loaded units listed.
The dmesg show some mount information:
[4634771.631686] new mount options do not match the existing superblock, will be ignored
[4634776.674913] new mount options do not match the existing superblock, will be ignored
[4634781.719080] new mount options do not match the existing superblock, will be ignored
[4634784.615539] systemd-fstab-generator[2008783]: Running in a container, ignoring fstab device entry for /dev/disk/by-id/dm-uuid-LVM-vLzixZWzTVUComSxsOuTeXk5Bfft0pwxuNeCu8fd5cqyOkbOXnVdSMwt1afab14J.
[4634784.615548] systemd-fstab-generator[2008783]: Running in a container, ignoring fstab device entry for /dev/disk/by-uuid/a903c899-d7bf-4a36-9d9b-90beecc892da.
[4634784.978525] loop7: detected capacity change from 0 to 295352
[4634786.761713] new mount options do not match the existing superblock, will be ignored
[4634787.100353] systemd-fstab-generator[2008890]: Running in a container, ignoring fstab device entry for /dev/disk/by-id/dm-uuid-LVM-vLzixZWzTVUComSxsOuTeXk5Bfft0pwxuNeCu8fd5cqyOkbOXnVdSMwt1afab14J.
[4634787.100363] systemd-fstab-generator[2008890]: Running in a container, ignoring fstab device entry for /dev/disk/by-uuid/a903c899-d7bf-4a36-9d9b-90beecc892da.
[4634787.513445] systemd-fstab-generator[2008930]: Running in a container, ignoring fstab device entry for /dev/disk/by-id/dm-uuid-LVM-vLzixZWzTVUComSxsOuTeXk5Bfft0pwxuNeCu8fd5cqyOkbOXnVdSMwt1afab14J.
[4634787.513455] systemd-fstab-generator[2008930]: Running in a container, ignoring fstab device entry for /dev/disk/by-uuid/a903c899-d7bf-4a36-9d9b-90beecc892da.
[4634787.972478] systemd-fstab-generator[2008968]: Running in a container, ignoring fstab device entry for /dev/disk/by-id/dm-uuid-LVM-vLzixZWzTVUComSxsOuTeXk5Bfft0pwxuNeCu8fd5cqyOkbOXnVdSMwt1afab14J.
[4634787.972487] systemd-fstab-generator[2008968]: Running in a container, ignoring fstab device entry for /dev/disk/by-uuid/a903c899-d7bf-4a36-9d9b-90beecc892da.
[4634788.409253] systemd-fstab-generator[2009035]: Running in a container, ignoring fstab device entry for /dev/disk/by-id/dm-uuid-LVM-vLzixZWzTVUComSxsOuTeXk5Bfft0pwxuNeCu8fd5cqyOkbOXnVdSMwt1afab14J.
[4634788.409263] systemd-fstab-generator[2009035]: Running in a container, ignoring fstab device entry for /dev/disk/by-uuid/a903c899-d7bf-4a36-9d9b-90beecc892da.
[4635157.932697] systemd-fstab-generator[2013237]: Running in a container, ignoring fstab device entry for /dev/disk/by-id/dm-uuid-LVM-vLzixZWzTVUComSxsOuTeXk5Bfft0pwxuNeCu8fd5cqyOkbOXnVdSMwt1afab14J.
[4635157.932706] systemd-fstab-generator[2013237]: Running in a container, ignoring fstab device entry for /dev/disk/by-uuid/a903c899-d7bf-4a36-9d9b-90beecc892da.
[4635158.265066] loop7: detected capacity change from 0 to 295352
[4635158.656602] systemd-fstab-generator[2013294]: Running in a container, ignoring fstab device entry for /dev/disk/by-id/dm-uuid-LVM-vLzixZWzTVUComSxsOuTeXk5Bfft0pwxuNeCu8fd5cqyOkbOXnVdSMwt1afab14J.
[4635158.656613] systemd-fstab-generator[2013294]: Running in a container, ignoring fstab device entry for /dev/disk/by-uuid/a903c899-d7bf-4a36-9d9b-90beecc892da.
[4635159.072524] systemd-fstab-generator[2013335]: Running in a container, ignoring fstab device entry for /dev/disk/by-id/dm-uuid-LVM-vLzixZWzTVUComSxsOuTeXk5Bfft0pwxuNeCu8fd5cqyOkbOXnVdSMwt1afab14J.
[4635159.072533] systemd-fstab-generator[2013335]: Running in a container, ignoring fstab device entry for /dev/disk/by-uuid/a903c899-d7bf-4a36-9d9b-90beecc892da.
[4635159.533573] systemd-fstab-generator[2013373]: Running in a container, ignoring fstab device entry for /dev/disk/by-id/dm-uuid-LVM-vLzixZWzTVUComSxsOuTeXk5Bfft0pwxuNeCu8fd5cqyOkbOXnVdSMwt1afab14J.
[4635159.533581] systemd-fstab-generator[2013373]: Running in a container, ignoring fstab device entry for /dev/disk/by-uuid/a903c899-d7bf-4a36-9d9b-90beecc892da.
[4635159.925815] systemd-fstab-generator[2013410]: Running in a container, ignoring fstab device entry for /dev/disk/by-id/dm-uuid-LVM-vLzixZWzTVUComSxsOuTeXk5Bfft0pwxuNeCu8fd5cqyOkbOXnVdSMwt1afab14J.
[4635159.925824] systemd-fstab-generator[2013410]: Running in a container, ignoring fstab device entry for /dev/disk/by-uuid/a903c899-d7bf-4a36-9d9b-90beecc892da.
[4636006.771317] systemd-fstab-generator[2020370]: Running in a container, ignoring fstab device entry for /dev/disk/by-id/dm-uuid-LVM-vLzixZWzTVUComSxsOuTeXk5Bfft0pwxuNeCu8fd5cqyOkbOXnVdSMwt1afab14J.
[4636006.771325] systemd-fstab-generator[2020370]: Running in a container, ignoring fstab device entry for /dev/disk/by-uuid/a903c899-d7bf-4a36-9d9b-90beecc892da.
[4636011.813695] new mount options do not match the existing superblock, will be ignored
[4636016.845525] new mount options do not match the existing superblock, will be ignored
[4636036.993422] new mount options do not match the existing superblock, will be ignored
[4636042.029244] new mount options do not match the existing superblock, will be ignored
[4636047.063301] new mount options do not match the existing superblock, will be ignored
[4636052.106122] new mount options do not match the existing superblock, will be ignored
[4636057.139267] new mount options do not match the existing superblock, will be ignored
[4636062.182795] new mount options do not match the existing superblock, will be ignored
[4636067.225208] new mount options do not match the existing superblock, will be ignored
I try to revert the lxd and upgrade it again, but it still failed:
root@qa-physical-5-4:~# snap changes
ID Status Spawn Ready Summary
76 Error today at 01:55 UTC today at 01:56 UTC Refresh "lxd" snap
77 Error today at 02:01 UTC today at 02:02 UTC Auto-refresh snap "lxd"
78 Done today at 02:16 UTC today at 02:16 UTC Running service command
79 Error today at 02:17 UTC today at 02:17 UTC Refresh "lxd" snap
80 Error today at 02:21 UTC today at 02:22 UTC Refresh "lxd" snap
81 Done today at 02:25 UTC today at 02:25 UTC Revert "lxd" snap
82 Error today at 02:25 UTC today at 02:26 UTC Refresh "lxd" snap
Sorry for reply slowly, the disk of host is healthy:
root@qa-physical-5-4:~# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 23G 3.0M 23G 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 196G 23G 164G 13% /
tmpfs 111G 64K 111G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup
tmpfs 111G 0 111G 0% /run/qemu
/dev/sdc2 974M 163M 745M 18% /boot
lxc-storage/kvm 1.7T 191G 1.6T 11% /data/kvm
tmpfs 1.0M 0 1.0M 0% /var/snap/lxd/common/ns
tmpfs 23G 4.0K 23G 1% /run/user/0
root@qa-physical-5-4:~# snap change 82
Status Spawn Ready Summary
Done yesterday at 02:25 UTC yesterday at 02:26 UTC Ensure prerequisites for "lxd" are available
Undone yesterday at 02:25 UTC yesterday at 02:26 UTC Download snap "lxd" (24323) from channel "latest/stable/ubuntu-21.10"
Done yesterday at 02:25 UTC yesterday at 02:26 UTC Fetch and check assertions for snap "lxd" (24323)
Undone yesterday at 02:25 UTC yesterday at 02:26 UTC Mount snap "lxd" (24323)
Undone yesterday at 02:25 UTC yesterday at 02:26 UTC Run pre-refresh hook of "lxd" snap if present
Undone yesterday at 02:25 UTC yesterday at 02:26 UTC Stop snap "lxd" services
Undone yesterday at 02:25 UTC yesterday at 02:26 UTC Remove aliases for snap "lxd"
Undone yesterday at 02:25 UTC yesterday at 02:26 UTC Make current revision for snap "lxd" unavailable
Error yesterday at 02:25 UTC yesterday at 02:26 UTC Copy snap "lxd" data
Hold yesterday at 02:25 UTC yesterday at 02:26 UTC Setup snap "lxd" (24323) security profiles
Hold yesterday at 02:25 UTC yesterday at 02:26 UTC Make snap "lxd" (24323) available to the system
Hold yesterday at 02:25 UTC yesterday at 02:26 UTC Automatically connect eligible plugs and slots of snap "lxd"
Hold yesterday at 02:25 UTC yesterday at 02:26 UTC Set automatic aliases for snap "lxd"
Hold yesterday at 02:25 UTC yesterday at 02:26 UTC Setup snap "lxd" aliases
Hold yesterday at 02:25 UTC yesterday at 02:26 UTC Run post-refresh hook of "lxd" snap if present
Hold yesterday at 02:25 UTC yesterday at 02:26 UTC Start snap "lxd" (24323) services
Hold yesterday at 02:25 UTC yesterday at 02:26 UTC Remove data for snap "lxd" (24175)
Hold yesterday at 02:25 UTC yesterday at 02:26 UTC Remove snap "lxd" (24175) from the system
Hold yesterday at 02:25 UTC yesterday at 02:26 UTC Clean up "lxd" (24323) install
Hold yesterday at 02:25 UTC yesterday at 02:26 UTC Run configure hook of "lxd" snap if present
Hold yesterday at 02:25 UTC yesterday at 02:26 UTC Run health check of "lxd" snap
Done yesterday at 02:25 UTC yesterday at 02:26 UTC Handling re-refresh of "lxd" as needed
......................................................................
Copy snap "lxd" data
2023-01-29T02:26:12Z ERROR cannot find installed snap "lxd" at revision 24323: missing file /snap/lxd/24323/meta/snap.yaml
root@qa-physical-5-4:~# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
lxc-storage 1.98T 425G 1.57T - - 36% 20% 1.00x ONLINE -
ssd 1.81T 438G 1.39T - - 41% 23% 1.00x DEGRADED -
ssd-benchmark 1.81T 467M 1.81T - - 3% 0% 1.00x ONLINE -
I’m wondering if a system restart may help out there, though keep in mind that this may also well result in whatever containers are on that system to not start back up until the snap issue is resolved…
It looks like snapd attempts to mount the snap at /snap/lxd/24323 but then that mount is empty for some reason. It could be an issue with the mount layout on the system masking that path, or it could be a bad .squashfs or issue with the kernel mounting it.
That’s good to hear. I’d recommend you report your issue at https://forum.snapcraft.io so hopefully the snapd team can track this down and make sure it doesn’t happen again…
?