Hello! I’m figuring out clustering, but there are some pieces of info that are missing, that would allow me to fully understand how clustering work. So, let’s assume I have a cluster of three servers, and there are few containers with routed network configured, with static, external IPs assigned. They have the same IPs on each member of the cluster. My question is: how the external router knows to which member of the cluster route packets? How is the “current” instance chosen, when we are looking from the pov of the outside router? I’m interested in understanding how it works 
Maybe there are some keywords that could help me to research this more?
For routed NICs, you generally need to have the instance’s IP be routed to the corresponding server for things to work, or potentially rely on proxy_arp to have it work more or less automatically.
The cluster doesn’t really do anything special for those NICs. Whatever server the instance is located on is the one that’s going to be expected to have that traffic routed to. How you achieve that is up to you.
Incus does have a built-in BGP server which may be useful in those situations, but I’ve never used it for routed NICs so I’m not sure that it will advertise the IP and a suitable next-hop.
Oh, so correct me if I’m wrong, but is it working like that:
Cluster decides instance on which server is “active”. It’s the only one running, the other ones are effectively stopped, but synced. If the “active” instance goes down, then the cluster decides which other one gets active. In the case of routed NIC we just need to wait a few minutes for the network to realize that the route to the IP changed, and that’s all? (so basically the same thing as what happens after incus move)
If that’s correct, I have a followup question:
Let’s say one of the members gets rebooted because it was updated or whatever. If it will be rebooting for let’s say 10 minutes, what will happen to the instance that was active on that server? In 10 minutes, will another instance take over? And then, after the rebooted member will be up again, will its previously active instance get synced, and will it become active once more automatically?
I’m just looking to understand how clustering works in practical terms .
No. An instance is assigned to a specific server, it doesn’t magically move around the cluster.
Look at incus list, the LOCATION column tells you where the instance is. If that server goes down so does that instance.
You can move the instance around by using incus move NAME --target NEW-LOCATION.
For a container you need to stop it first though, for VMs, you can do it live but need things configured to handle that.
So if you create instance A and it gets created on server B, server B is the one that needs the instance IP routed to it. If you then decide to move instance A to server C, then now server C is the one that needs the instance traffic routed to it. All other servers don’t do anything for instance A.
1 Like
Oh, so the only thing that’s redundant is configuration (database) and the cluster itself serves only a purpose of load balancing? I grossly misunderstood it then ^^" , but now I think I got it. Thanks!