What if some but not all LXD cluster hosts need additional storage

LXD
1

Hi

We usually have a host or two in certain sites that might have additional storage, particularly for backups to a SAN or DAS, e.g. lxdhost3 below. On those hosts we’d have one of the software clustered services, like a Docker management only node, MySQL slave, ArangoDB follower, Domino cluster member… etc (in VMware VMs at the mo). These services on the “backup” host still participate in quorum elections. These nodes tend to have additional disks in them as well.

With a LXD cluster it appears must have pools (ZFS) the same across all hosts, so for disk0, 1, 2 in the e.g. below that’s fine, but when it comes to lxdhost3, we now have disk3 or more on other hosts in another domain.

Without breaking the LXD cluster so that lxdhost3 has it’s own storage pools, would:

  1. the next best approach be to create a pool with loop/dir backed sources on lxdhost1 & 2 which won’t be used, then a proper disk backed source on lxdhost3 for use with containers/vms on that lxdhost3,
  2. is there a better approach,
  3. and am I missing some concepts?

Seems to have worked but I’m in test environment at the mo.

E.g.:

lxdhost1

  • disk0 = SAS RAID10, primary OS files, app binaries
  • disk1 = NVMe, data
  • disk2 = NVMe, data

lxdhost2

  • disk0 = SAS RAID10, primary OS files, app binaries
  • disk1 = NVMe, data
  • disk2 = NVMe, data

lxdhost3

  • disk0 = SAS RAID10, primary OS files, app binaries
  • disk1 = SAS RAID10, data
  • disk2 = SAS RAID10, data
  • disk3 = iSCSI-SAN, backups

LXD Cluster Storage Pools

  • lxd_local_disk0 = lxdhost1.disk0, lxdhost2.disk0, lxdhost3.disk0
  • lxd_local_disk1 = lxdhost1.disk1, lxdhost2.disk1, lxdhost3.disk1
  • lxd_local_disk2 = lxdhost1.disk2, lxdhost2.disk2, lxdhost3.disk2
  • lxd_san_backups1 = lxdhost1.loop, lxdhost2.loop, lxdhost3.disk3 (shared by multiple containers/VMs)

Thanks

2

You’re absolutely correct, LXD currently expects identical storage pool and network setup on all servers, so to deal with your requirement, you’d indeed find yourself creating a tiny loop-based pool on the first two servers and use a real disk on the 3rd.

I realize it’s a bit silly and I think it’d a design requirement of LXD clusters that while making our life significantly easier (especially for evacuation, instance migration, …) we may have to make a bit more flexible in the future.