Do all LXD Cluster member nodes have to be on the same LAN

I read thru:

But don’t think I saw anything about this referenced.

I currently use my own Wireguard VPN Mesh to interconnect at Layer 2, multiple LXD servers & their Containers/VMs across multiple nodes on multiple Clouds (Hetzner & Digital Ocean).

But I was thinking about @tomp work with LXD/VxLAN/OVN and wondered if there’s been any integration with that and LXD Clusters ?

Seems like LXD Clusters could then have member Nodes inter-cloud, inter-Datacenters ?

Just thought I’d ask.

Thanks

LXD clusters themselves do not need to be in the same subnet. Although we do not recommend doing WAN clusters (>5ms of latency between them) as dqlite requires low latency between members.

Normal bridge networks are created on each cluster member and use SNAT of the host’s external address, so they would be fine being in different subnets.

The fan network overlay requires the cluster members to be in the same subnet.

For OVN, the actual OVN overlay doesn’t require cluster members be in the same subnet.
However it is important to understand that in LXD each cluster can act as a potential ingress/egress point to the uplink network for all of the instances in that network (i.e there is only one active router gateway chassis in the network at a time).

This normally means that OVN uplink networks need to be in the same L2 subnet.
However it is not strictly true if you are only doing egress through the uplink and are doing SNAT on the host, and don’t mind the SNAT address changing depending on which cluster member chassis is active for that network.

See What is topology of lxd in creating ovn logical switches and router? - #9 by tomp

Thanks for the details Tom.

I can understand the latency problem considerations.

But inter-cloud and intra-cloud nodes often have upto 10Gbps bw so perhaps that would stay w/in LXD Cluster requirements?

I could try it w my current Wireguard Mesh VPN.

I’d really like to also see about using the LXD/VxLAN/OVN solution as an alternative approach to the Wireguard Mesh VPN :blush:

Thanks for the background!

Brian

Latency (time for a packet to travel between the members) is the key metric as opposed to bandwidth (the data transfer rate) of the interconnect.

See https://github.com/lxc/lxd/issues/9861 for some of the sorts of issues you may experience if latency is too high.

1 Like