How to setup OVN for cluster

1

I have following:

3 Mini-PC’s with 1 ethernet port
Setup incus cluster with OVN the details are following:

MIni PC 1:
IPv4 address: 192.168.1.12

Mini PC 2:
IPv4 address: 192.168.1.13

Mini PC 3:
IPv4 address: 192.168.1.14

Now to setup cluster did following:

Setup netplan with br0 the setup is as follows for netplan:

network:
  version: 2
  renderer: networkd
  ethernets:
    enp1s0:
      dhcp4: no
      dhcp6: no
  bridges:
    br0:
      dhcp4: no
      dhcp6: no
      addresses:
        - 192.168.1.12/24
        - 172.21.8.1/24
      routes:
        - to: default
          via: 192.168.1.1
      interfaces:
        - enp1s0
      nameservers:
        search: [example.com]
        addresses: [1.1.1.1, 1.1.2.2, 8.8.8.8]

For the other two just change the ip but to 13,14 for 192 subnet and 2 and 3 for 172 subnet.
In all the PC’s ip_forward is set to 1 using sysctl.

Now created an ovn cluster for this 3 PC’s.

After setting up ovn cluster properly did the following to setup the network:

incus network create UPLINK --type=physical parent=br0 --target=pc1
incus network create UPLINK --type=physical parent=br0 --target=pc2
incus network create UPLINK --type=physical parent=br0 --target=pc3
incus network create UPLINK --type=physical    ipv4.ovn.ranges=172.21.8.6-172.21.8.88    ipv4.gateway=172.21.8.1/24    dns.nameservers=1.1.1.1,1.1.2.2

incus network create test-ovn --type=ovn
incus launch images:ubuntu/22.04 c1 --network test-ovn

incus list
+------+---------+---------------------+-----------------------------------------------+-----------+-----------+----------+
| NAME |  STATE  |        IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS | LOCATION |
+------+---------+---------------------+-----------------------------------------------+-----------+-----------+----------+
| c1   | RUNNING | 10.133.131.2 (eth0) | fd42:483b:c318:8d60:216:3eff:fe3d:4e36 (eth0) | CONTAINER | 0         | pc1      |
+------+---------+---------------------+-----------------------------------------------+-----------+-----------+----------+

But the container c1 cannot access internet or even the internal address of 192.168.1.12 or 13.

If I create an UPLINK within 192.168.10/24 network with gateway of 192.168.1.1 then it can.

incus network create UPLINK --type=physical    ipv4.ovn.ranges=192.168.1.190-192.168.1.206    ipv4.gateway=192.168.1.1/24    dns.nameservers=1.1.1.1,1.1.2.2

I do not want to use 192.168.1.1 subnet as its not controlled by me for the ovn network and thats the reason choose another ip of 172.21.8.0/24 network for it. But 172.21.8.0 network is not directly connected to internet as using PC1 as the gateway node with ip 172.21.8.1, but it doesn’t seem to work form within the ovn network probably some setup linked in ovn-nbctl for router needs to be updated.

Anyone can let me know how to fix it?

2

Your 172.21.8.1/24 is only valid on that one system it’s on, the other two systems in your cluster have no idea how to reach that and if they all have that same address, then you have a MAC address conflict on your network which is similarly going to be a problem.

Even if that somehow worked, those systems are probably not configured to forward that traffic out and if they did, your upstream router (192.168.1.1) would have no idea how to route traffic back to 172.21.8.X.

So this really isn’t a setup that’s going to work. You need your actual router (192.168.1.1) to either run multiple different subnets on that physical network, at which point your router would become 172.21.8.1/24 and would let you put whatever you want on 172.21.8.X/24.

Or you need to configure that router to exclude a range of address from its DHCP (say 192.168.1.200-192.168.1.254) and then provide that as the OVN range in Incus.

3

No nodes have same address the other two nodes have 172.21.8.2 and 172.21.8.3. Example I put is only node for 1 node out of 3 as other two just differs in IP address.

Also I can ping 172.21.8.1 from the other two nodes. So network is connected and bridge br0 has 2 addresses one in 192.168.1.0/24 network and another in 172.21.8.0/24 network.

I thought since ip_forward is setup in 192.168. 1.12 network. Any packets destined for internet from 172.21.8.1 will be automatically handled as default gateway 192.168.1.1 from that machine is reachable.

4

Okay, that’s still a bad idea as 172.21.8.1 is only available on one of the three servers, so when you shut down or reboot that machine, it won’t be available anymore.

Did you configure your host to NAT all outgoing traffic towards 192.168.1.x?
if you didn’t, then your router at 192.168.1.x will be getting traffic coming from 172.21.8.x and have absolutely no idea what that is or how to send responses back.

5

Yes thats true as I did not setup distributed gateway using ovn.

Also physical network router is single point of failure if not redundant. Since its a home network thought its alright.

I thought I don’t need to NAT 172.x traffic as default gateway is set as 192.168.1.1 but traffic might not be handled from there as there is no 172.21/subnet there.

So I need to setup IP masquerade from 172.x to 192.168, probably that might be the reason why it’s happening. So two times masquerade 10.x network to 172.21 and from 172.21 to 192.168.

6

Ok finally was able to fix the problem the whole setup works now with following process:

Use netplan to setup br0 on 3 machines

PC1:
network:
  version: 2
  renderer: networkd
  ethernets:
    enp1s0:
      dhcp4: no
      dhcp6: no
  bridges:
    br0:
      dhcp4: no
      dhcp6: no
      addresses:
        - 192.168.1.12/24
        - 172.21.8.1/24
      routes:
        - to: default
          via: 192.168.1.1
      interfaces:
        - enp1s0
      nameservers:
        search: [example.com]
        addresses: [1.1.1.1, 1.1.2.2, 8.8.8.8]

PC2:
network:
  version: 2
  renderer: networkd
  ethernets:
    enp1s0:
      dhcp4: no
      dhcp6: no
  bridges:
    br0:
      dhcp4: no
      dhcp6: no
      addresses:
        - 192.168.1.13/24
        - 172.21.8.2/24
      routes:
        - to: default
          via: 192.168.1.1
      interfaces:
        - enp1s0
      nameservers:
        search: [example.com]
        addresses: [1.1.1.1, 1.1.2.2, 8.8.8.8]

PC3:
network:
  version: 2
  renderer: networkd
  ethernets:
    enp1s0:
      dhcp4: no
      dhcp6: no
  bridges:
    br0:
      dhcp4: no
      dhcp6: no
      addresses:
        - 192.168.1.14/24
        - 172.21.8.3/24
      routes:
        - to: default
          via: 192.168.1.1
      interfaces:
        - enp1s0
      nameservers:
        search: [example.com]
        addresses: [1.1.1.1, 1.1.2.2, 8.8.8.8]

Now created a 172.21.8.0 based ovn uplink physical network

incus network create UPLINK --type=physical parent=br0 --target=PC1
incus network create UPLINK --type=physical parent=br0 --target=PC2
incus network create UPLINK --type=physical parent=br0 --target=PC3
incus network create UPLINK --type=physical    ipv4.ovn.ranges=172.21.8.8-172.21.8.28    ipv4.gateway=172.21.8.1/24    dns.nameservers=1.1.1.1,1.1.2.2 ovn.ingress_mode=routed ipv4.routes=192.168.1.0/24,172.21.8.0/24 
incus network create test-ovn --type=ovn
incus launch images:ubuntu/22.04 c1 --network test-ovn
+------+---------+--------------------+----------------------------------------------+-----------+-----------+----------+
| NAME |  STATE  |        IPV4        |                     IPV6                     |   TYPE    | SNAPSHOTS | LOCATION |
+------+---------+--------------------+----------------------------------------------+-----------+-----------+----------+
| c1   | RUNNING | 10.26.199.2 (eth0) | fd42:150:3e38:224d:216:3eff:fe7d:1cf4 (eth0) | CONTAINER | 0         | PC1      |
+------+---------+--------------------+----------------------------------------------+-----------+-----------+----------+
incus exec c1 -- bash
root@c1:~# ping www.yahoo.com
^C
root@c1:~# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
^C
--- 192.168.1.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2048ms

Now after setting up a iptables rules in PC1 it begins to work.

sudo iptables -t nat -A POSTROUTING -o br0 -s 172.21.8.0/24 -j SNAT --to-source 192.168.1.168
incus exec c1 -- bash
root@c1:~# ping www.yahoo.com
PING me-ycpi-cf-www.g06.yahoodns.net (106.10.236.40) 56(84) bytes of data.
64 bytes from o2.ycpi.vip.sg3.yahoo.com (106.10.236.40): icmp_seq=1 ttl=46 time=5.48 ms
64 bytes from o2.ycpi.vip.sg3.yahoo.com (106.10.236.40): icmp_seq=2 ttl=48 time=4.51 ms
64 bytes from o2.ycpi.vip.sg3.yahoo.com (106.10.236.40): icmp_seq=3 ttl=48 time=3.34 ms
^C
--- me-ycpi-cf-www.g06.yahoodns.net ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 3.337/4.442/5.477/0.875 ms
root@c1:~# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=253 time=2.19 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=253 time=0.369 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=253 time=0.364 ms
^C
--- 192.168.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2009ms
rtt min/avg/max/mdev = 0.364/0.973/2.186/0.857 ms

@stgraber Thanks for pointing out the issue now its working, the drawback as you mentioned earlier in this approach is PC1 is single point of failure as gateway.