Looking for a little help with OVN and creating an UPLINK

Hi, I’ve been trying to work with the documentation to set up Incus with OVN on a cluster. It mostly seems to work but I seem to be falling at the last hurdle. i.e. creating the UPLINK network.
Incus UI / cluster is showing all nodes up, ovs seems consistent across all three nodes, so it feels like it’s working as intended …

# ovs-vsctl show
f8096055-1b18-4ee3-ae5f-ffdc9873f4b8
    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port ovn-abd377-0
            Interface ovn-abd377-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.10.10.3"}
        Port ovn-3c321d-0
            Interface ovn-3c321d-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.10.10.2"}
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "3.1.0"

I have a bridge set up which both Incus and OVN are using;

# ifconfig br1
br1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.10.1  netmask 255.0.0.0  broadcast 10.10.10.255
        inet6 2a00:23c7:3c21:cc01:6e6e:7ff:fe16:a2f7  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::6e6e:7ff:fe16:a2f7  prefixlen 64  scopeid 0x20<link>
        ether 6c:6e:07:16:a2:f7  txqueuelen 1000  (Ethernet)
        RX packets 70199  bytes 18206614 (17.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1104773  bytes 197194883 (188.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

So if I do;

incus network create UPLINK --type=physical parent=br1 --target=core
incus network create UPLINK --type=physical parent=br1 --target=borg
incus network create UPLINK --type=physical parent=br1 --target=p400
incus network create UPLINK --type=physical 
incus network set UPLINK ipv4.ovn.ranges=10.10.10.200-10.10.10.254 

This seems to complete and UPLINK appears in the UI.
But, on the Overview page for UPLINK, the UI reports;

Could not load network state: Network interface "UPLINK" not found

Despite everything else saying it was “CREATED”;

# incus network list
+----------------+----------+---------+------+------+-------------+---------+---------+
|      NAME      |   TYPE   | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY |  STATE  |
+----------------+----------+---------+------+------+-------------+---------+---------+
| UPLINK         | physical | YES     |      |      |             | 0       | CREATED |
+----------------+----------+---------+------+------+-------------+---------+---------+
| br0            | bridge   | NO      |      |      |             | 0       |         |
+----------------+----------+---------+------+------+-------------+---------+---------+
| br1            | bridge   | NO      |      |      |             | 2       |         |
+----------------+----------+---------+------+------+-------------+---------+---------+
| br-int         | bridge   | NO      |      |      |             | 0       |         |
+----------------+----------+---------+------+------+-------------+---------+---------+
| eth0           | physical | NO      |      |      |             | 0       |         |
+----------------+----------+---------+------+------+-------------+---------+---------+
| eth1           | physical | NO      |      |      |             | 0       |         |
+----------------+----------+---------+------+------+-------------+---------+---------+
| genev_sys_6081 | unknown  | NO      |      |      |             | 0       |         |
+----------------+----------+---------+------+------+-------------+---------+---------+
| lo             | loopback | NO      |      |      |             | 0       |         |
+----------------+----------+---------+------+------+-------------+---------+---------+
| ovs-system     | unknown  | NO      |      |      |             | 0       |         |
+----------------+----------+---------+------+------+-------------+---------+---------+
| wlan0          | physical | NO      |      |      |             | 0       |         |
+----------------+----------+---------+------+------+-------------+---------+---------+

And when I move on to create an OVN network, it gives me what I take to be a partially Bogus error that I’m guessing is the result of the network interface not actually having been created;

# incus network create my-ovn --type=ovn
Error: Failed generating auto config: Failed to automatically find an unused IPv4 subnet, manual configuration required

I’ve looked at all the logging I can find and I can’t seem to see what looks like an error message anywhere. Anyone any idea what might be wrong or where I can look to see why it doesn’t want to actually create UPLINK?

Working with Raspberry Pi’s, Raspberry Pi OS (Debian) self compiled kernel 6.6.78 kernel with Geneve module enabled. Incus version 6.10.1.

tia

Looks like you might have missed a step. Suggest to review and look as this old thread OVN high availability cluster tutorial. There hasn’t much changed except you need to replace LXC with Incus and you should get a working OVN setup.

Hmm. I’ve written a repeatable script that strips back the DB’s on all machines and installs (as per the instructions) all nodes at the same time.
On completion all the nodes look good, logging indicates that Raft is happy.

… except that it yields the same result when I try to create the OVN network.
I am completing all the documented steps, all the tests I can see indicate that both the Incus and OVN clusters are working fine. Creating the uplink goes through with no Errors … yet it would appear something isn’t working, and that something isn’t obvious from the logs.
If anyone has any ideas of where to look I would be very grateful …

# ./ovn.sh uplink
Network UPLINK pending on member core
Network UPLINK pending on member borg
Network UPLINK pending on member p400
Network UPLINK created
* Configuring NorthBound

All fine …

# incus network create my-ovn --type=ovn
Error: Failed generating auto config: Failed to automatically find an unused IPv4 subnet, manual configuration required

I’m not familiar with OVN (other than this experience) but I was half expecting this output to contain something resulting from the creation of UPLINK. IF this is the case, does this imply the Incus code that creates the OVN network device is failing silently?

# ovs-vsctl show 
205d76b6-a69a-453c-b560-ed31f09fd08f
    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port ovn-abd377-0
            Interface ovn-abd377-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.10.10.3"}
        Port br-int
            Interface br-int
                type: internal
        Port ovn-3c321d-0
            Interface ovn-3c321d-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.10.10.2"}
    ovs_version: "3.1.0"

Ok, so looking at “incus monitor”, IF I have missed a step or done something wrong, it feels like I shouldn’t be seeing then when UPLINK is created;

location: core
metadata:
  context:
    config: map[dns.nameservers:192.168.1.254 ipv4.gateway:192.168.1.254/24 ipv4.ovn.ranges:192.168.1.16-192.168.1.31
      parent:br0 volatile.last_state.created:false]
    member: p400
    network: UPLINK
    project: default
  level: debug
  message: Created network on cluster member
timestamp: "2025-03-18T17:44:57.388460153Z"
type: logging


location: core
metadata:
  context:
    config: map[dns.nameservers:192.168.1.254 ipv4.gateway:192.168.1.254/24 ipv4.ovn.ranges:192.168.1.16-192.168.1.31
      parent:br0 volatile.last_state.created:false]
    member: borg
    network: UPLINK
    project: default
  level: debug
  message: Created network on cluster member
timestamp: "2025-03-18T17:44:57.536554507Z"
type: logging


location: core
metadata:
  context:
    network: UPLINK
    project: default
  level: debug
  message: Marked network global status as created
timestamp: "2025-03-18T17:44:57.560899149Z"
type: logging

So it’s saying it’s created, AND it seems to be reporting that other nodes are acknowleding it at created too … yet the UI reports CREATED, but at the same time;

Could not load network state: Network interface "UPLINK" not found

I must admit to being a little exasperated / confused … to all intents and purposes it says it’s doing everything successfully, but at the same time it’s not working. Incidentally, I’ve tried with a bunch of different address combinations, this was me seeing whether DHCP routed addresses for the UPLINK variables would make any difference.

I’d tear it all down and try from scratch, but the script I’m now using does this every time I try. I’m assuming;

(stop ovn)
     rm -f /var/lib/ovn/*
     rm -r /var/lib/openvswitch/*
(start ovn)

Is enough for a full reset …

Which of these network interfaces is the one created by incus init? On default it is named “incusbr0” but it seems like you have choosen a different name.
The default Incus network name is the one you should use to create your OVN network against and use it as the “UPLINK”. In short instead of using

incus network create UPLINK --type=physical parent=br1 --target=core

try to use:

incus network create br1 --target=core

Assume br1 is your uplink interface. This is how it is described in OVN high availability cluster tutorial. OVN need to know which is the uplink interface / network of your host. There is no such “UPLINK” interface on your host which results into the error:

Assigning the correct network interface during OVN network creation will solve the error.

That is at least the difference I can spot.

Hi, thanks for the reply, but I’m not sure the Tutorial you’re referencing applies in context. It seems to be for OVN between VM’s, whereas I’m working on a bare metal setup using;

inuxcontainers.org/incus/docs/main/howto/network_ovn_setup/

Specifically; when you create an uplink (in the UI for example) it only offers you the option of using unused physical devices, which I’m guessing is why;

incus network create UPLINK --type=physical parent=br1 --target=core

Is the documented approach (that I’m seeing), i.e. it creates a physical device (which it seems to need) from the bridge that has the required connectivity.

The cluster was initially set up on br1, all the networks listed in the table were created automatically by Incus, with the exception of UPLINK, which was created as per the instructions in the link listed … so “incus network create br1 --target=core” isn’t going to do anything, and I’m guessing removing br1 isn’t going to work with two references against it.

In general there is no difference between a VM or bare metal, except VM’s are virtualized. All other features are exactly the same and comparable in terms of command line options.

Would suggest to start completely from scratch by following the thread I posted. This is how I setup my cluster using default settings for incus and ovn. This guide is confirmed to work but there are for sure other ways how to get it working…

Hi, thanks, but “just follow these docs because it works this way” isn’t something I can use in production. I’ve followed the docs, I’ve written scripts to automatically tear down and rebuild as per the docs, and as for me, it’s “failing silently”. If there is no way of debugging this or understanding the difference between working and non-working configs, it would not seem wise to progress. I’ve been working with ipv4 for over 30 years, if I’m not getting this now then it’s probably too late for me.

I’m going to step back for now and just use bridges, maybe I’ll try again in the future after a more younger experts have a trodden path to follow … :slight_smile:
Many thanks.

Hmm. So … I tore down all three servers and rebuilt them without OVN.
Still had a problem with bridges (!)

Turns out Incus seems to have an issue with using addresses in the 10.0.0.0/8 range for interface IP addresses. Now this sounds completely mad, however. I switched my 10.10.10.0/24 ranges out for 192.168.2.0/24 ranges, and hey presto, my non-OVN setup worked.

I’ve now re-applied all my OVN code, and guess what - it works!

There are a couple of references online to people seeing a similar issue, here’s one;

Note the error quoted here is the one I was experiencing when trying to create an OVN network. Now I’m a little bit speechless and somewhat embarrassed to post this, because it can’t be right that the issue is actually the non-routed address range chosen … and I will update it if I actually find I did something else wrong, but for now, as far as I can see, switching 10.10.10.0/24 for 192.168.2.0/24 fixed the problem. (My OVN/Incus deployment is mostly automated, so I’m reasonably confident this is the only change)