I’m having a number of issues trying to restart my cluster after a provider-forced reboot
The most pressing appears to be that one node is waiting for others to upgrade, but they in turn aren’t running and can’t be removed from the cluster either.
I’m going around in circles here, so could do with pointing in the right direction.
~# lxd sql global "SELECT * FROM nodes;"
+----+---------+-------------+-------------------------------------+--------+----------------+--------------------------------+---------+------+
| id | name | description | address | schema | api_extensions | heartbeat | pending | arch |
+----+---------+-------------+-------------------------------------+--------+----------------+--------------------------------+---------+------+
| 1 | cloud10 | | 172.16.0.10:8443 | 30 | 194 | 2020-06-25T14:09:08.481184632Z | 0 | 2 |
| 2 | cloud11 | | 172.16.11.1:8443 | 30 | 194 | 2021-02-17T14:12:42.566365351Z | 0 | 2 |
| 3 | cloud14 | | cloud14.internal.wrm-media.com:8443 | 44 | 230 | 2021-02-17T14:16:53.541032426Z | 0 | 2 |
| 4 | cloud13 | | cloud13.internal.wrm-media.com:8443 | 44 | 230 | 2021-02-17T14:16:53.541237655Z | 0 | 2 |
| 5 | cloud12 | | cloud12.internal.wrm-media.com:8443 | 30 | 194 | 2021-02-17T12:59:48.637427932Z | 0 | 2 |
+----+---------+-------------+-------------------------------------+--------+----------------+--------------------------------+---------+------+
cloud10 and 11 are currently broken, so I want to try and push past them if possible.
cloud13 appears to be the main database node, but it’s repeating this in it’s log:
t=2021-02-17T14:16:37+0000 lvl=eror msg="Unaccounted raft node(s) not found in 'nodes' table for heartbeat: map[0:{ID:1 Address:0 Role:spare} 172.16.14.1:8443:{ID:3 Address:172.16.14.1:8443 Role:spare}]"
172.16.14.1 points to cloud14, so I’m not sure why that’s showing up, and I don’t know what the other entry in that map is referring to.