Hi
I installed 3 nodes for microceph and it works fine. I was looking however to increase the speed of the storage, because it is noticeable slower than local storage.
I increased the network to 10Gb which helps a bit, and I also added more nodes, 4 and 5. I noticed however that the monitors and managers stay locked to 3, even if you have more nodes. Is it possible to increase the quorum to 4 or 5 then? ceph-deploy is not available in microceph and ceph orch always gives me a “Error ENOENT: No orchestrator configured (try ceph orch set backend)”, but trying to set a backend doesn’t seem to help me.
Increasing the number of monitors past 3 only really makes sense if you have dozens of server, otherwise the increase will just slow things down.
You also need your quorum to be an uneven number or you’re going to get into potential split brain type situations.
The way the monitor service works is that it’s a shared database where each change must be written to all others prior to the change being considered committed and things moving on. Increasing the number of server therefore typically comes with less performance, not more. The actual work isn’t spread across all monitors either, instead an election process happens and one is elected the leader. The leader is the one performing the vast majority of the work, with the others just being there to keep a copy of the data as it gets written.
Hi Stéphane, thank you for clearing this out.
So, the reason why I would suggest to be able to have more monitors is because I noticed some speed performances when deliberately shutting down one of the servers. This gave me a 25% speed boost with a 5 node setup (which becomes 4 then). I noticed, if I use only 3 nodes the speed is slower than the 5 node and I wanted to examine if I reduce the 5 node cluster by disabling 2 nodes what the speed would compare to the original 3 node. Of course if you have a quorum of 1 then, this cannot be tested. If the 5 node setup would have 5 monitors, 2 nodes can fail as well, if you only have 3 monitors only 1 can fail. So, that would also be of a great benefit.
Thanks again for putting out this great LXD eco system!