Hi! I need some help.
yesterday, our 2 servers in the cluster upgraded themselves to version 5.6. Although tracking set only on latest/stable, I did a snap revert. now LXD does not start with message in logs:
Error: Failed to initialize global database: failed to ensure schema: this node's version is behind, please upgrade
network connectivity is fine, other servers in cluster has version 5.5, that server have the same version now.
You’re running the non-LTS latest/stable snap channel.
For LXD this means you cannot downgrade after an upgrade (only the LTS series allow for downgrades).
For LXD clusters all cluster members need to be on the same snap revision.
So the way forward is to perform a sudo snap refresh lxd on all cluster members.
LXD has built-in logic that detects when one cluster member has been updated by snap and will then ask the other cluster members to perform a snap refresh too, so as to minimize the time between to all cluster members being on the same version.
However snap has this concept of cohorts and phasing, which can mean that depending on how the cluster was originally installed, some members may not be in the same cohort and will not see the update straight away.
Or there could be another reason why some member(s) are not being upgraded.
Either way you should perform a snap refresh on each member and then if the cluster doesn’t recover please can you get the output of the LXD daemon logs on each cluster member, as well as the output of sudo snap info lxd.
For more information on managing when the snap performs its updates see:
Thanks
but how it possible?
so now I must switch all other servers to latest/candidate? why only 2 server upgraded to this wrong channel?
Please show full output of snap info lxd from each member.
Cutted description, first 2 is working fine, 3rd working(afraid to do something with this server), last one not working.
name: lxd
summary: LXD - container and VM manager
publisher: Canonical✓
store-url: https://snapcraft.io/lxd
contact: https://github.com/lxc/lxd/issues
license: unset
services:
lxd.activate: oneshot, enabled, inactive
lxd.daemon: simple, enabled, active
lxd.user-daemon: simple, enabled, inactive
snap-id: J60k4JY0HppjwOjW8dZdYc8obXKxujRu
tracking: latest/stable
refresh-date: 32 days ago, at 15:14 UTC
channels:
latest/stable: 5.5-37534be 2022-08-27 (23537) 113MB -
latest/candidate: 5.6-794016a 2022-09-23 (23680) 142MB -
latest/beta: ↑
latest/edge: git-c76bc42 2022-09-25 (23689) 142MB -
5.6/stable: –
5.6/candidate: 5.6-794016a 2022-09-23 (23680) 142MB -
5.6/beta: ↑
5.6/edge: ↑
5.5/stable: 5.5-37534be 2022-08-27 (23537) 113MB -
5.5/candidate: 5.5-37534be 2022-08-19 (23537) 113MB -
5.5/beta: ↑
5.5/edge: ↑
5.4/stable: 5.4-1ff8d34 2022-08-13 (23367) 108MB -
5.4/candidate: 5.4-3bf11b7 2022-08-12 (23453) 108MB -
5.4/beta: ↑
5.4/edge: ↑
5.3/stable: 5.3-91e042b 2022-07-06 (23270) 107MB -
5.3/candidate: 5.3-b403e7f 2022-07-05 (23282) 107MB -
5.3/beta: ↑
5.3/edge: ↑
5.0/stable: 5.0.1-9dcf35b 2022-08-24 (23541) 107MB -
5.0/candidate: 5.0.1-9dcf35b 2022-08-19 (23541) 107MB -
5.0/beta: ↑
5.0/edge: git-13e1e53 2022-08-21 (23564) 113MB -
4.0/stable: 4.0.9-8e2046b 2022-03-26 (22753) 71MB -
4.0/candidate: 4.0.9-8e2046b 2022-03-24 (22753) 71MB -
4.0/beta: ↑
4.0/edge: git-407205d 2022-03-31 (22797) 73MB -
3.0/stable: 3.0.4 2019-10-10 (11348) 55MB -
3.0/candidate: 3.0.4 2019-10-10 (11348) 55MB -
3.0/beta: ↑
3.0/edge: git-81b81b9 2019-10-10 (11362) 55MB -
installed: 5.5-37534be (23537) 113MB in-cohort
name: lxd
summary: LXD - container and VM manager
publisher: Canonical✓
store-url: https://snapcraft.io/lxd
contact: https://github.com/lxc/lxd/issues
license: unset
services:
lxd.activate: oneshot, enabled, inactive
lxd.daemon: simple, enabled, active
lxd.user-daemon: simple, enabled, inactive
snap-id: J60k4JY0HppjwOjW8dZdYc8obXKxujRu
tracking: latest/stable
refresh-date: 32 days ago, at 15:15 UTC
channels:
latest/stable: 5.5-37534be 2022-08-27 (23537) 113MB -
latest/candidate: 5.6-794016a 2022-09-23 (23680) 142MB -
latest/beta: ↑
latest/edge: git-c76bc42 2022-09-25 (23689) 142MB -
5.6/stable: –
5.6/candidate: 5.6-794016a 2022-09-23 (23680) 142MB -
5.6/beta: ↑
5.6/edge: ↑
5.5/stable: 5.5-37534be 2022-08-27 (23537) 113MB -
5.5/candidate: 5.5-37534be 2022-08-19 (23537) 113MB -
5.5/beta: ↑
5.5/edge: ↑
5.4/stable: 5.4-1ff8d34 2022-08-13 (23367) 108MB -
5.4/candidate: 5.4-3bf11b7 2022-08-12 (23453) 108MB -
5.4/beta: ↑
5.4/edge: ↑
5.3/stable: 5.3-91e042b 2022-07-06 (23270) 107MB -
5.3/candidate: 5.3-b403e7f 2022-07-05 (23282) 107MB -
5.3/beta: ↑
5.3/edge: ↑
5.0/stable: 5.0.1-9dcf35b 2022-08-24 (23541) 107MB -
5.0/candidate: 5.0.1-9dcf35b 2022-08-19 (23541) 107MB -
5.0/beta: ↑
5.0/edge: git-13e1e53 2022-08-21 (23564) 113MB -
4.0/stable: 4.0.9-8e2046b 2022-03-26 (22753) 71MB -
4.0/candidate: 4.0.9-8e2046b 2022-03-24 (22753) 71MB -
4.0/beta: ↑
4.0/edge: git-407205d 2022-03-31 (22797) 73MB -
3.0/stable: 3.0.4 2019-10-10 (11348) 55MB -
3.0/candidate: 3.0.4 2019-10-10 (11348) 55MB -
3.0/beta: ↑
3.0/edge: git-81b81b9 2019-10-10 (11362) 55MB -
installed: 5.5-37534be (23537) 113MB in-cohort
name: lxd
summary: LXD - container and VM manager
publisher: Canonical✓
store-url: https://snapcraft.io/lxd
contact: https://github.com/lxc/lxd/issues
license: unset
services:
lxd.activate: oneshot, enabled, inactive
lxd.daemon: simple, enabled, active
lxd.user-daemon: simple, enabled, inactive
snap-id: J60k4JY0HppjwOjW8dZdYc8obXKxujRu
tracking: latest/stable
refresh-date: yesterday at 17:19 UTC
channels:
latest/stable: 5.5-37534be 2022-08-27 (23537) 113MB -
latest/candidate: 5.6-794016a 2022-09-23 (23680) 142MB -
latest/beta: ↑
latest/edge: git-c76bc42 2022-09-25 (23689) 142MB -
5.6/stable: –
5.6/candidate: 5.6-794016a 2022-09-23 (23680) 142MB -
5.6/beta: ↑
5.6/edge: ↑
5.5/stable: 5.5-37534be 2022-08-27 (23537) 113MB -
5.5/candidate: 5.5-37534be 2022-08-19 (23537) 113MB -
5.5/beta: ↑
5.5/edge: ↑
5.4/stable: 5.4-1ff8d34 2022-08-13 (23367) 108MB -
5.4/candidate: 5.4-3bf11b7 2022-08-12 (23453) 108MB -
5.4/beta: ↑
5.4/edge: ↑
5.3/stable: 5.3-91e042b 2022-07-06 (23270) 107MB -
5.3/candidate: 5.3-b403e7f 2022-07-05 (23282) 107MB -
5.3/beta: ↑
5.3/edge: ↑
5.0/stable: 5.0.1-9dcf35b 2022-08-24 (23541) 107MB -
5.0/candidate: 5.0.1-9dcf35b 2022-08-19 (23541) 107MB -
5.0/beta: ↑
5.0/edge: git-13e1e53 2022-08-21 (23564) 113MB -
4.0/stable: 4.0.9-8e2046b 2022-03-26 (22753) 71MB -
4.0/candidate: 4.0.9-8e2046b 2022-03-24 (22753) 71MB -
4.0/beta: ↑
4.0/edge: git-407205d 2022-03-31 (22797) 73MB -
3.0/stable: 3.0.4 2019-10-10 (11348) 55MB -
3.0/candidate: 3.0.4 2019-10-10 (11348) 55MB -
3.0/beta: ↑
3.0/edge: git-81b81b9 2019-10-10 (11362) 55MB -
installed: 5.6-794016a (23680) 142MB in-cohort
name: lxd
summary: LXD - container and VM manager
publisher: Canonical✓
store-url: https://snapcraft.io/lxd
contact: https://github.com/lxc/lxd/issues
license: unset
services:
lxd.activate: oneshot, enabled, inactive
lxd.daemon: simple, enabled, inactive
lxd.user-daemon: simple, enabled, inactive
snap-id: J60k4JY0HppjwOjW8dZdYc8obXKxujRu
tracking: latest/stable
refresh-date: today at 07:04 UTC
channels:
latest/stable: 5.5-37534be 2022-08-27 (23537) 113MB -
latest/candidate: 5.6-794016a 2022-09-23 (23680) 142MB -
latest/beta: ↑
latest/edge: git-c76bc42 2022-09-25 (23689) 142MB -
5.6/stable: –
5.6/candidate: 5.6-794016a 2022-09-23 (23680) 142MB -
5.6/beta: ↑
5.6/edge: ↑
5.5/stable: 5.5-37534be 2022-08-27 (23537) 113MB -
5.5/candidate: 5.5-37534be 2022-08-19 (23537) 113MB -
5.5/beta: ↑
5.5/edge: ↑
5.4/stable: 5.4-1ff8d34 2022-08-13 (23367) 108MB -
5.4/candidate: 5.4-3bf11b7 2022-08-12 (23453) 108MB -
5.4/beta: ↑
5.4/edge: ↑
5.3/stable: 5.3-91e042b 2022-07-06 (23270) 107MB -
5.3/candidate: 5.3-b403e7f 2022-07-05 (23282) 107MB -
5.3/beta: ↑
5.3/edge: ↑
5.0/stable: 5.0.1-9dcf35b 2022-08-24 (23541) 107MB -
5.0/candidate: 5.0.1-9dcf35b 2022-08-19 (23541) 107MB -
5.0/beta: ↑
5.0/edge: git-13e1e53 2022-08-21 (23564) 113MB -
4.0/stable: 4.0.9-8e2046b 2022-03-26 (22753) 71MB -
4.0/candidate: 4.0.9-8e2046b 2022-03-24 (22753) 71MB -
4.0/beta: ↑
4.0/edge: git-407205d 2022-03-31 (22797) 73MB -
3.0/stable: 3.0.4 2019-10-10 (11348) 55MB -
3.0/candidate: 3.0.4 2019-10-10 (11348) 55MB -
3.0/beta: ↑
3.0/edge: git-81b81b9 2019-10-10 (11362) 55MB -
installed: 5.5-37534be (23537) 113MB -
OK so the third member is now on 5.6 whilst the others are on 5.5. This is a problem.
Have you run sudo snap refresh lxd on the other 3 members to see if they will come up to LXD 5.6?
it answer: snap “lxd” has no updates available
because channel is to stable
In terms of what you’re seeing regarding the latest/stable channel showing as 5.5 but having 5.6 installed, I think this is actually a snap bug with the way they do phased roll outs.
Its pretty confusing I agree.
I don’t understand how it happens, why only 2 servers make update from other channel, that they have on them setting.
so now, you advice set all members to latest/candidate and make update?
Try running sudo snap switch lxd --cohort=+ on all systems followed by sudo snap refresh lxd
I opened this snap issue to see if the snap info output can be improved:
thank you! now all online. How can we switch to that LTS latest/stable snap channel?
Did running sudo snap refresh lxd --cohort="+" on all systems work?
The LTS series channels are named like <version>/stable e.g. the current LTS series is 5.0/stable (see Managing the LXD snap).
Because you cannot downgrade from a feature release, you equally cannot downgrade back to an LTS release. So you would need to reinstall LXD from scratch using an LTS channel, such as sudo snap install lxd --channel=5.0/stable.
will I loose my configuration after that? such as containers for example?
Did running sudo snap refresh lxd --cohort="+" on all systems work?
Because if you can get it back up you can export your instances in order to import them back into a fresh LTS installation.
no, I just made snap refresh lxd, and it get updated
So LXD cluster is working again now?
Can you provide snap info lxd for each member so I can check they are all in the cohort.
Yes, you would need to backup (lxc export) your instances first so they can be reimported later.