Clustering guidance (VXLAN, forward, or proxy)

LXD
,
1

Host= Ubuntu 20.04 LTS
LXD (snap) = 5.0.1
Using ZFS storage pools on all hosts

My use case, I have a multiple VLAN network with 3 LXD hosts on the same (management) VLAN (layer 2 network), each host is statically addressed. I have all three hosts configured with a managed bridge (br0) to provide network abstraction, (as I can’t have all my containers on the same (macvlan) layer 2 network without issues of duplicate IPs) for security reasons. All VLANs are routed on the LAN and to the ISP uplink interface. The containers use static addresses as well. In the past I have used proxy devices in the container config for forwarding ingress traffic to containers running web apps (NGINX reverse-proxy, apache, email) especially when running in a public VPS, with great success. My goal with this project is to be able to migrate these web containers from one host to the other without re-addressing the containers. At the moment all three hosts use the same addressing scheme on the bridge (10.148.4.0/24), so the IP addresses move around without issue. All hosts are on the same VLAN, the containers currently just NAT the outbound traffic with “ipv4.nat: “true”” on the managed bridge.

My question is: can I use a network forward “floating IP” to push inbound/outbound traffic to/from the managed bridge by port number, then on to the containers? I have also considered VXLAN (multicast with tunnel.lan.protocol: vxlan) since the hosts all share the same L2 network, but I like to keep things as simple as possible. Either way I’d like to utilize the features in LXD where I can for simplicity. I am not using any routing protocols, no BGP, no OVS - just tagged interfaces at the host level and I let the edge router handle all the routing/filtering from the WAN. Is there any advantage with one over the other, between proxy , VXLAN, or floating IP forwards to the managed bridge?

Thank you,
Happy Holidays everyone!

2

Doing a bit of experimenting with network forwarding, and not sure any of this is correct or ideal, but here is what is working for me in a lab setup to test out the various options for networking. Opinions? Criticisms?

I have my hosts setup with tagged interfaces on a vlan dedicated to container instances (VLAN50).

Here is the netplan config on the first lab host:

network:
  version: 2
  renderer: NetworkManager
  ethernets:
      enp2s0:
          dhcp4: no
  vlans:
      VLAN50:
          id: 50
          link: enp2s0
          addresses: [192.168.50.10/24]
          nameservers:
              addresses: [192.168.50.1, 1.1.1.1]
      VLAN50:
          id: 50
          link: enp2s0
          addresses: [192.168.50.15/24]
          nameservers:
              addresses: [192.168.50.1, 1.1.1.1]
# Management interface
      VLAN95:
          id: 95
          link: enp2s0
          addresses: [192.168.0.12/24]
          nameservers:
              addresses: [192.168.0.1, 1.1.1.1]
          routes:
             - to: default
               via: 192.168.0.1

Other hosts are identical, with different addresses on vlan50.

I then have my test forward setup to listen on 50.10 for initial testing:

lxc network forward ls br0
+----------------+-------------+------------------------+-------+
| LISTEN ADDRESS | DESCRIPTION | DEFAULT TARGET ADDRESS | PORTS |
+----------------+-------------+------------------------+-------+
| 192.168.50.10  |             | 10.148.4.90            | 0     |
+----------------+-------------+------------------------+-------+

For the purposes of this testing, I just have the default target to send all ports to the test instance on 10.148.4.90. I will separate this out by port if I move this into production cluster at some point.

Here is the managed bridge network:

lxc network show br0
config:
  ipv4.address: 10.148.4.1/24
  ipv4.nat: "true"
  ipv6.address: none
  ipv6.nat: "true"
description: ""
name: br0
type: bridge

I installed nginx on the test instance just to test with curl. Here is the config of the test instance:

devices:
  eth0:
    ipv4.address: 10.148.4.90
    nictype: bridged
    parent: br0
    type: nic

With this configuration I can access the external network from the container, and I can curl the container forwarded address (192.168.50.10) from anywhere on the network with success. I have its sister hosts addressed much the same with 50.20 and 25 forwarded to the same instance.

My goal with this setup again, is to be able to move containers between hosts in the cluster, without changing the instance config (ip4 address). I am using different IP’s on the other hosts in the same VLAN, that I can then load balance from the gateway.

As always, open to criticism or alternative suggestions.

Thanks!

3

I am testing network forwards with a tunnel configured between the hosts in multicast. This seems to work quite well so far in my testing from the bridge addresses in the containers.

I have one lab host setup as:

test@nuc1:~$ lxc network show lxdbrx
config:
  ipv4.address: 10.191.210.1/24
  ipv4.nat: "true"
  ipv6.address: none
  ipv6.nat: "true"
  tunnel.lan.id: "10"
  tunnel.lan.protocol: vxlan
description: ""
name: lxdbrx
type: bridge
used_by:
- /1.0/instances/u1
- /1.0/instances/u2
- /1.0/profiles/vxlan
managed: true

My other host has the same setup:

test@300U-i3:~$ lxc network show lxdbrx
config:
  ipv4.address: 10.191.210.1/24
  ipv4.nat: "true"
  ipv6.address: none
  ipv6.nat: "true"
  tunnel.lan.id: "10"
  tunnel.lan.protocol: vxlan
description: ""
name: lxdbrx
type: bridge
used_by:
- /1.0/instances/u1
- /1.0/instances/u2
- /1.0/profiles/vxlan
managed: true

Both hosts are on the same L2 network to make multicast work, both bridges share the same address segment, if one container goes down on one host, both hosts are still able to ping/communicate with the other instance on the sister host and vice versa.

My next step is to test connectivity from the forwarded addresses in vlan50. I will report back with my findings, and see if this works for my use case.

So far looking like I need a combination of forwarded addresses, and a tunnel to keep everything consistent and connected. Sort of a simplified HA setup at this point.