Lxc cluster list show all 5 servers DATABASE NO

LXD
1

Everything seems to be running fine, but a lxc cluster list show all servers online but none with Database. Client & Server version: 3.18
Again everything seems to be fine, but obviously something is wrong. Any ideas?

2

Please can you paste the output of:

lxd sql global "SELECT * FROM nodes"

and

lxd sql global "SELECT * FROM nodes_roles"

?

It feels that the nodes_roles table in the database is not populated correctly.

3

±—±---------±------------±-----------------±-------±---------------±------------------------------------±--------+
| id | name | description | address | schema | api_extensions | heartbeat | pending |
±—±---------±------------±-----------------±-------±---------------±------------------------------------±--------+
| 1 | MOE | | 64.71.77.32:8443 | 18 | 148 | 2019-12-11T08:16:22.679948906-05:00 | 0 |
| 2 | LARRY | | 64.71.77.80:8443 | 18 | 148 | 2019-12-11T08:16:22.680222798-05:00 | 0 |
| 3 | JOE | | 64.71.77.13:8443 | 18 | 148 | 2019-12-11T08:16:22.680446058-05:00 | 0 |
| 4 | CHEMP | | 64.71.77.18:8443 | 18 | 148 | 2019-12-11T08:16:22.680609322-05:00 | 0 |
| 5 | CURLYJOE | | 64.71.77.29:8443 | 18 | 148 | 2019-12-11T08:16:22.680765875-05:00 | 0 |
±—±---------±------------±-----------------±-------±---------------±------------------------------------±--------+

lxd sql global “SELECT * FROM nodes_roles”
±--------±-----+
| node_id | role |
±--------±-----+
±--------±-----+

`

4

This is all working fine, and working fine since last problem. But Obviously something is wrong.

5

What’s the output of:

lxd sql local "SELECT * FROM raft_nodes"

?

6

@stgraber it seems that the nodes_roles table is not populated, unless you have better ideas I’m going to provide to @Tony_Anytime some manual INSERT queries to run to create those entries. However it’s unclear how we got to this state.

7

lxd sql local “SELECT * FROM raft_nodes”
±—±-----------------+
| id | address |
±—±-----------------+
| 1 | 64.71.77.32:8443 |
| 2 | 64.71.77.80:8443 |
| 4 | 64.71.77.13:8443 |
±—±-----------------+

8

Ok, please run:

lxd sql global "INSERT INTO nodes_roles(node_id, role) VALUES (1, 0)"
lxd sql global "INSERT INTO nodes_roles(node_id, role) VALUES (2, 0)"
lxd sql global "INSERT INTO nodes_roles(node_id, role) VALUES (3, 0)"

that should fix the problem.

9

That’s fine, though it’s puzzling why the table would have gotten wiped…

10

Yes this fixed the problem. Thanks

11

Just FYI: I’ve been creating a cluster node-by-node, and ran into the same issue after a couple of node reboots. My guess is that I had a node freak out, lose connection to the other two nodes, and then it never synced up again.

12

Before you give up try this in all you machine
make yourself a resetlxd.sh

echo ‘Stopping Socket’
systemctl stop snap.lxd.daemon.unix.socket
sleep 10
echo ‘Starting Unix Socket’
systemctl start snap.lxd.daemon.unix.socket
sleep 10
echo ‘Stopping lxd’
systemctl stop snap.lxd.daemon
echo ‘Starting lxd’
systemctl restart snap.lxd.daemon
sleep 10
lxc cluster list

Let me know if it works for you/

13

Thanks much, but the server in question had greater underlying issues than its cluster membership. But I’ll snippet that for the future, thanks!