Why not allow cluster nodes to start that are updated though?
So long as quorum is achieved anyhow.
When the out of date node tries to participate in the cluster reject the request, with a need to update response and perhaps even trigger a refresh.
On the lxc end when a user issues a command on the out of date node maybe add a message that indicates the node needs to be updated when receiving the need to update response or even when selected as a target on a remote node.
If a node that is removed from the cluster when it is offline and comes back online, send a response that indicates it is not a cluster member. On the lxc end perhaps prompt if the user would like to lxd init on the node or rejoin the cluster if it is of a compatible version, if it is not a compatible version indicate that the node would need to be updated to rejoin.
Obviously if you remove the node and had containers running on it they would need to be retargeted in a distributed manner to the rest of the nodes. I didn’t have any containers running on the down node so unsure if this happens already or not.
I suppose that would only apply if there was shared storage, in the instance that it is not shared storage if/when the down node comes back online convert it to a standalone lxd instance. So the containers that were stored locally are not lost.
Additionally, when issuing lxc commands in a waiting state unless yall decide to implement the above perhaps give a lxc response indicating “Waiting for all nodes to upgrade schema…” so we aren’t left wondering what is happening with our cluster without killing all container instances to start in debug mode.
Along that same line, maybe allow for toggling debug mode while lxd is running. lxc --enable-local-debug & lxc --disable-local-debug | lxc --toggle-local-debug.
Alright, done editing for now =). The above should increase stability, responsiveness and availability of clusters. One of these days I will get around to learning Go so I can actually contribute, stuck deep in Python and c++ projects for now.