Error while setting up OVN for cluster

Incus
1

Hello,

I am setting up an OVN for cluster latest Incus version 6.0. I have 4 nodes as follows:

Node 1:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether 2c:cf:67:0a:f0:f3 brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 2c:cf:67:0a:f0:f4 brd ff:ff:ff:ff:ff:ff
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a2:dd:cd:a2:96:00 brd ff:ff:ff:ff:ff:ff
5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 66:62:2a:d7:75:15 brd ff:ff:ff:ff:ff:ff
6: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether 2e:97:f3:f7:8e:e7 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9c7b:70ff:fec2:87a3/64 scope link 
       valid_lft forever preferred_lft forever
7: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ee:f7:fd:27:dd:01 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.179/24 brd 10.0.1.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::ecf7:fdff:fe27:dd01/64 scope link 
       valid_lft forever preferred_lft forever

Node 2: 10.0.1.176
Node 3: 10.0.1.178
Node 4: 10.0.1.177

I get an error when creating UPLINK network on Node 1 as shown below:

incus network create UPLINK --type=physical parent=br0 --target=rpicluster01
Network UPLINK pending on member rpicluster01
root@rpicluster01:~# incus network create UPLINK --type=physical parent=br0 --target=rpicluster02
Network UPLINK pending on member rpicluster02
root@rpicluster01:~# incus network create UPLINK --type=physical parent=br0 --target=rpicluster03
Network UPLINK pending on member rpicluster03
root@rpicluster01:~# incus network create UPLINK --type=physical parent=br0 --target=rpicluster04
Network UPLINK pending on member rpicluster04
root@rpicluster01:~# incus network create UPLINK --type=physical
Error: failed to notify peer 10.0.1.176:8443: Parent interface "br0" not found
root@rpicluster01:~#

Here’s the configuration I put into /etc/netplan config file:

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: no
  bridges:
    br0:
      interfaces: [eth0]
      addresses: [10.0.1.179/24]
      routes:
        - to: default
          via: 10.0.1.1
      nameservers:
        addresses:
          - 1.1.1.1
          - 1.0.0.1
      parameters:
        stp: true
        forward-delay: 4
      dhcp4: no

Output from incus network ls is as follows:

root@rpicluster01:/etc/netplan# incus network ls
+--------+----------+---------+------+------+-------------+---------+---------+
|  NAME  |   TYPE   | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY |  STATE  |
+--------+----------+---------+------+------+-------------+---------+---------+
| UPLINK | physical | YES     |      |      |             | 0       | ERRORED |
+--------+----------+---------+------+------+-------------+---------+---------+
| br0    | bridge   | NO      |      |      |             | 0       |         |
+--------+----------+---------+------+------+-------------+---------+---------+
| eth0   | physical | NO      |      |      |             | 0       |         |
+--------+----------+---------+------+------+-------------+---------+---------+
| wlan0  | physical | NO      |      |      |             | 0       |         |
+--------+----------+---------+------+------+-------------+---------+---------+

What am I doing wrong?

2

It suggests that the machine with IP 10.0.1.176 doesn’t have a br0 interface

3

I see. So should all nodes have br0 configured? Assuming that’s what you meant.

4

Yep, for what you’re trying to do above, you need every server to have a br0 interface.

5

Please forgive the naive questions (this is completely new to me: what IP address should I assign to each br0 interface? Would it be some random static IP address within my range, or would it be one of the nodes?

6

shouldn’t really matter if it’s just a bridge going to an external physical network you don’t even really need an IP on there

7

The only reason for assigning an IP address to br0 is if you want to use it as a management address for the node itself.

If it only exists to carry guest traffic, it’s better to leave off any IP (v4 or v6), so that it doesn’t become a way to attack the host.

In Netplan, using manual bridges rather than OVN, I do this:

network:
  version: 2
  ethernets:
    enp1s0:
      dhcp4: false
      accept-ra: false
      link-local: []
  vlans:
    vlan251:
      id: 251
      link: enp1s0
      accept-ra: false
      link-local: []
    vlan252:
      id: 252
      link: enp1s0
      accept-ra: false
      link-local: []
    # etc
  bridges:
    br251:
      macaddress: XX:XX:XX:XX:XX:XX
      interfaces: [vlan251]
      parameters:
        stp: false
        forward-delay: 0
      dhcp4: false
      accept-ra: false
      link-local: []
    br252:
      macaddress: XX:XX:XX:XX:XX:XX
      interfaces: [vlan252]
      parameters:
        stp: false
        forward-delay: 0
      dhcp4: false
      accept-ra: false
      link-local: []
    # etc
2 Likes