Error while setting up OVN for cluster

Hello,

I am setting up an OVN for cluster latest Incus version 6.0. I have 4 nodes as follows:

Node 1:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether 2c:cf:67:0a:f0:f3 brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 2c:cf:67:0a:f0:f4 brd ff:ff:ff:ff:ff:ff
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a2:dd:cd:a2:96:00 brd ff:ff:ff:ff:ff:ff
5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 66:62:2a:d7:75:15 brd ff:ff:ff:ff:ff:ff
6: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether 2e:97:f3:f7:8e:e7 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9c7b:70ff:fec2:87a3/64 scope link 
       valid_lft forever preferred_lft forever
7: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ee:f7:fd:27:dd:01 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.179/24 brd 10.0.1.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::ecf7:fdff:fe27:dd01/64 scope link 
       valid_lft forever preferred_lft forever

Node 2: 10.0.1.176
Node 3: 10.0.1.178
Node 4: 10.0.1.177

I get an error when creating UPLINK network on Node 1 as shown below:

incus network create UPLINK --type=physical parent=br0 --target=rpicluster01
Network UPLINK pending on member rpicluster01
root@rpicluster01:~# incus network create UPLINK --type=physical parent=br0 --target=rpicluster02
Network UPLINK pending on member rpicluster02
root@rpicluster01:~# incus network create UPLINK --type=physical parent=br0 --target=rpicluster03
Network UPLINK pending on member rpicluster03
root@rpicluster01:~# incus network create UPLINK --type=physical parent=br0 --target=rpicluster04
Network UPLINK pending on member rpicluster04
root@rpicluster01:~# incus network create UPLINK --type=physical
Error: failed to notify peer 10.0.1.176:8443: Parent interface "br0" not found
root@rpicluster01:~# 

Here’s the configuration I put into /etc/netplan config file:

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: no
  bridges:
    br0:
      interfaces: [eth0]
      addresses: [10.0.1.179/24]
      routes:
        - to: default
          via: 10.0.1.1
      nameservers:
        addresses:
          - 1.1.1.1
          - 1.0.0.1
      parameters:
        stp: true
        forward-delay: 4
      dhcp4: no

Output from incus network ls is as follows:

root@rpicluster01:/etc/netplan# incus network ls
+--------+----------+---------+------+------+-------------+---------+---------+
|  NAME  |   TYPE   | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY |  STATE  |
+--------+----------+---------+------+------+-------------+---------+---------+
| UPLINK | physical | YES     |      |      |             | 0       | ERRORED |
+--------+----------+---------+------+------+-------------+---------+---------+
| br0    | bridge   | NO      |      |      |             | 0       |         |
+--------+----------+---------+------+------+-------------+---------+---------+
| eth0   | physical | NO      |      |      |             | 0       |         |
+--------+----------+---------+------+------+-------------+---------+---------+
| wlan0  | physical | NO      |      |      |             | 0       |         |
+--------+----------+---------+------+------+-------------+---------+---------+

What am I doing wrong?

It suggests that the machine with IP 10.0.1.176 doesn’t have a br0 interface

I see. So should all nodes have br0 configured? Assuming that’s what you meant.

Yep, for what you’re trying to do above, you need every server to have a br0 interface.

Please forgive the naive questions (this is completely new to me: what IP address should I assign to each br0 interface? Would it be some random static IP address within my range, or would it be one of the nodes?

shouldn’t really matter if it’s just a bridge going to an external physical network you don’t even really need an IP on there

The only reason for assigning an IP address to br0 is if you want to use it as a management address for the node itself.

If it only exists to carry guest traffic, it’s better to leave off any IP (v4 or v6), so that it doesn’t become a way to attack the host.

In Netplan, using manual bridges rather than OVN, I do this:

network:
  version: 2
  ethernets:
    enp1s0:
      dhcp4: false
      accept-ra: false
      link-local: []
  vlans:
    vlan251:
      id: 251
      link: enp1s0
      accept-ra: false
      link-local: []
    vlan252:
      id: 252
      link: enp1s0
      accept-ra: false
      link-local: []
    # etc
  bridges:
    br251:
      macaddress: XX:XX:XX:XX:XX:XX
      interfaces: [vlan251]
      parameters:
        stp: false
        forward-delay: 0
      dhcp4: false
      accept-ra: false
      link-local: []
    br252:
      macaddress: XX:XX:XX:XX:XX:XX
      interfaces: [vlan252]
      parameters:
        stp: false
        forward-delay: 0
      dhcp4: false
      accept-ra: false
      link-local: []
    # etc
1 Like