We are in the process of moving instances from a LXD cluster to an Incus cluster and eventually decommission the hardware. There are three cluster members.
We are now down to only two instances on that LXD cluster. The instances occupy 300GB storage.
Would it be at all possible to untangle the cluster and have the three cluster members survive as a single node LXD systems? And most importantly, can we keep the two instances alive?
You can’t turn a cluster into a bunch of standalone servers.
What you can do is move the instances onto a single server in the cluster and then remove the other servers from the cluster, effectively keeping a cluster of one server running your instances.
But can you turn a single-node cluster back into a single non-clustered node? That is, is it possible to reverse incus cluster enable ?
No, the database change from standalone to cluster is irreversible. But a one node cluster will generally act close enough to a standalone system.
The main difference is that its cluster IP address must always be present on the system (but you can add it to the loopback device as a workaround) and that the cluster database tasks will be running so the CPU won’t be able to idle as much as it does on standalone.
Both of those make for an annoying experience of you accidentally cluster a laptop but aren’t usually a big deal for a server.
Thanks. What if you reinstall incus from scratch and then do incus admin recover? Is there anything in the container state which would cause clustering to be re-enabled?
That would work, but make sure to manually save anything else you may have as the disaster recovery will only get you back your instances and volumes, it has limited support for profiles but doesn’t know about server config, projects, networks and the like, so those you may want to save as YAML.
I would suggest that it would be good if you could cluster or uncluster at will. I understand the present technical reasons/limitations. but as a user that has had managed and dealt with clusters in a real world working environment flexibility is key.
It’s pretty unlikely that we’ll do that. It would require a bunch of work within the underlying database engine (cowsql) which would make it even more complex than it already is. Or we’d need to effectively have dedicated logic in Incus to create a blank database and move all the data across, mangling records as needed, merging split records (network, storage and volumes), …
That’d be quite a lot of logic which would be used extremely infrequently and typically used in environments that are already not in the best shape, so the likelihood of something going wrong and make things even worse would be rather high.
I think it’s much simpler to just say (as is done today) that clustering is a one way switch. If you want to undo it, get your data out and reinstall, that way you’ll have a nice and consistent install again.
I know it is hard… But make a poll and ask users how they feel. Last time, I ran into a problem I spoke with a several very knowledgeable people in the forums a couple told me they would not implement a cluster because it locks them in. Again suggestion, ask the user base.
If someone in the community feels like investing the weeks/months to change cowsql and Incus to allow safely unclustering and can do so with automated testing to prevent regressions and testing to show no performance impact or regression for those not using clustering, we’d be happy to review it.
I would suggest that it would be good if you could cluster or uncluster at will. I understand the present technical reasons/limitations. but as a user that has had managed and dealt with clusters in a real world working environment flexibility is key.
I’d just like to add that, as a user, this sounds like a terrible prioritization of developer time over everything else that could be improved. Surely 99% of clusters that are set up are set up with the explicit purpose of being just that, clusters. If for some reason you’d want to convert to a non-clustered incus setup, what would “unclustering” provide that manual migration of containers and vms with incus export does not already?
A little more feedback about this un-clustering.
What I noticed is that the LXD cluster cannot run on a single node. It needs to have at least two nodes alive. So my initial plan wouldn’t work anyway.
Meanwhile I moved all my remaining containers to a new (Incus) cluster. I am going to dismantle the LXD cluster nodes and maybe revive them as new nodes in the Incus cluster.
I have been running Lxd/Incus clusters for years…. and they break. I have lost the whole cluster several times. Just search the forum under my name. It is much reliable today, but if you have a hardware failure and have to run your cluster with half your machines and then have to repair and bring one back online, it does not help if all your clients are down. There are many people here who wont run a cluster because they are afraid of loosing the whole thing. Again, it is much better now, but I would still like the ability to uncluster and recluster at will.
I haven’t used clusters much, but if I were to use them, I would setup a test environment (i.e. three Incus VMs and inside there each of the three cluster members). I would figure out some way to debug and know what’s happening behind the scenes. If something were to break, I would need ways for troubleshooting. I would break small things on purpose in order to figure out how that looks like in the logs, and then how to fix.
Considering the latest pinned post, Incus clusters are the way to go. Stéphane explained that there’s a possibility to perform the conversion programmatically (if someone undertakes the task) or you can just move the assets to a new (non-clustered) Incus installation.
I have been using INCUS cluster since day one, they can be great or a real pain depending on why it gets stuck. I have in the past lost all my data for 30 containers and had to rebuild from backups several times. It is much much better now. But like a car that you can check the transmission oil level or change it, not being able to uncluster a cluster will one day leave stuck, broken and unable to fix it. Been there , done that too many times. I do use them in production environments. And I always keep at least 4 servers running so if something fails the cluster does not loses it ability to be managed…. Again it is much much better now.