What are the cases when it’s good/better to use LXD cluster?
For example, I have few servers with different number of disks each (all different sizes). I have to use local drives due to specific production load and 1gb network. I use nictype=routed for all containers with public ipv4. I do often migrate containers between physical nodes to rebalance CPU/memory/disk load so I have to do lxc remote add ... for each server to each, which requires manual work or some external scripting/orchestration.
Can I have benefits for this scenario if I’ll setup LXD cluster? Is it possible at all in case of different disks number? From How to configure storage for a cluster - LXD documentation - “All members of a cluster must have identical storage pools”, which is possible for me only if I’ll do zfs stripe on all disks in each node which is bad in terms of data loss.
Are there any other ways to optimize LXD setup for described scenario?
I believe that they just have to all have the same storage pools, but they can be backed by devices unique to that server.
Had the same issue with a storage server with a completely different storage layout that would run some instances, but in the end there was no real reason for it to be part of the cluster. Only ended up with some additional work with LXC client trusts and LXD profile configuration updates. I did bump into this issue recently though which had worked before, but this only affects custom storage volumes that I’m aware of and not actual instance transfers, when copying a custom volume from a cluster member to the stand-alone server using lxc storage volume copy {storagepool/volume} {standaloneserverremote:storagepool/volume}:
My original thought to get around storage pools on the storage server not being present on other cluster members, was to use loopback or directory backed storage pools which wouldn’t get used on the main cluster members that didn’t have the necessary physical devices… bit of a bad idea but gets around the requirement. Just needed to remember not to use them on the main cluster members, so perhaps establishing a good naming convention for this solution would be a little better so that you don’t start using them on servers that don’t have appropriate storage devices. There is also LVM that you could provide chunks of storage to back pools that don’t exist.
There are 2 ideas but I’d avoid the second one. Hopefully someone could provide better ideas. Also depends on what you need, and the above could be useless to you.
The thing I was not able to find in google and github issues is why lxd cluster was made as it is made now?
May be there is good reason for this and I’m using it wrong way?
Or may be it’s hard to implement flexible lxd cluster for all scenarios and current way is just most common used and I should do my own for different scenario?
I thought about making fake minimal size storage volumes for servers which have less disks than other servers but it looks like ugly solution and it will be hard to support as disks can be randomly added to servers.
When creating a storage pool on a cluster, yes, each member must have a pool of the same name and type. But the source property used can be different on each member.
So I am assuming you want a ZFS pool as you mentioned ZFS. LXD supports using an existing zpool or ZFS dataset for its source. So you can manually create the zpool/top-level dataset however you wish first and then instruct LXD to use it.
Servers are very different from different years. They all have different number of different disks.
In my case I can not change those servers to something new and expect new servers will be the same (and sometimes they will have upgrade like more memory and more disks to empty slots).
If I always do manually select storage to move container on any lxc move and in case I have different number of disks (and storages) per server - is it ok to make the first storage the same name on any server and have “fake” near zero size loop-based storages just to match the requirements of LXD clustering?
What should I do If I have new server with even more disks? Can I add new “fake” storage on all cluster members online or I should stop cluster someway before it or what?