LXD cluster limitations and usage scenario

Hi Nick

I believe that they just have to all have the same storage pools, but they can be backed by devices unique to that server.

Had the same issue with a storage server with a completely different storage layout that would run some instances, but in the end there was no real reason for it to be part of the cluster. Only ended up with some additional work with LXC client trusts and LXD profile configuration updates. I did bump into this issue recently though which had worked before, but this only affects custom storage volumes that I’m aware of and not actual instance transfers, when copying a custom volume from a cluster member to the stand-alone server using lxc storage volume copy {storagepool/volume} {standaloneserverremote:storagepool/volume}:
https://github.com/lxc/lxd/issues/11061

My original thought to get around storage pools on the storage server not being present on other cluster members, was to use loopback or directory backed storage pools which wouldn’t get used on the main cluster members that didn’t have the necessary physical devices… bit of a bad idea but gets around the requirement. Just needed to remember not to use them on the main cluster members, so perhaps establishing a good naming convention for this solution would be a little better so that you don’t start using them on servers that don’t have appropriate storage devices. There is also LVM that you could provide chunks of storage to back pools that don’t exist.

There are 2 ideas but I’d avoid the second one. Hopefully someone could provide better ideas. Also depends on what you need, and the above could be useless to you.