Limiting scope of lvmcluster to a subset of nodes

Incus
1

I wonder if there is a way to create an lvmcluster storage which is only accessible to a subset of nodes?

Here’s a specific example of what I might want to do:

  • Create a three node cluster: A, B, C
  • Create a partition on A and B with DRBD mirroring between them, configured as dual-primary
  • Create lvmcluster on top of that DRBD mirror device

A VM running on node A or B could access the shared volume with lvmcluster, and could live-migrate between those nodes. But the volume would not be accessible on node C.

I know there are cluster groups, and I could create a “drbd” group containing instances A and B to constraint an instance launch, but I don’t know if that prevents the instance being migrated to node outside its initial group - and I don’t think it prevents explicitly creating an instance on node C using the configured lvmcluster storage.

Extending this scenarios, you could also set up other mirrored pairs between B and C, and A and C, with their own separate storage pools, and VMs would need similar mobility constraints. I can also think of other scenarios where a particular SAN backend is only available to a subset of nodes, but you might still want all nodes to be in the same incus cluster.

Thanks!

2

No, currently it’s expected that all clustered servers share the same network and storage configuration.

This is something we’ve been considered to change but it gets EXTREMELY complex and so far has not been worth the time and risks.

1 Like
3

Thanks. I think it should be possible with drbd9, which allows N-way replication (up to 32 copies) and also allows disks to be accessed from nodes which don’t host a replica.

4

We’re planning on adding Linstor support in the near future so that would effectively offer a more flexible, more cloud-y DRBD

5

Ack. I was asked by a friend if it was possible to run incus over drbd directly, as he thinks linstor looks a bit too complex. I think that by the time you’ve set up drbd manually, it’s probably not a huge difference, especially since it takes cluster LVM and lvm-lockd/sanlock out of the equation.

I’ve just built a test cluster using three Ubuntu 22.04 + drbd9 nodes (Linbit haven’t published deb packages for drdb9 for 24.04 yet) and it does work, e.g. VMs can live migrate. I should probably write it up.

1 Like