OVN isn't currently available

Hello,
I am having trouble setting up an OVN cluster.
I got the following message in the logs:

time="2024-06-23T17:31:13+02:00" level=error msg="Failed shutting down network, couldn't load network" err="OVN isn't currently available" network=ovn-net project=default

The OVN cluster seems to be “properly” configured. Is there any more research I could add to find out where I’m wrong?

Here are the facts:

  • Cluster
incus cluster ls
+-------+--------------------------+-----------------+--------------+-----------------+-------------+--------+-------------------+
|  NOM  |           URL            |      ROLES      | ARCHITECTURE | DOMAINE D'ÉCHEC | DESCRIPTION | STATUS |      MESSAGE      |
+-------+--------------------------+-----------------+--------------+-----------------+-------------+--------+-------------------+
| home0 | https://192.0.0.200:9443 | database-leader | x86_64       | default         |             | ONLINE | Fully operational |
|       |                          | database        |              |                 |             |        |                   |
+-------+--------------------------+-----------------+--------------+-----------------+-------------+--------+-------------------+
| home1 | https://192.0.0.201:9443 | database        | x86_64       | default         |             | ONLINE | Fully operational |
+-------+--------------------------+-----------------+--------------+-----------------+-------------+--------+-------------------+
| home2 | https://192.0.0.202:9443 | database        | x86_64       | default         |             | ONLINE | Fully operational |
+-------+--------------------------+-----------------+--------------+-----------------+-------------+--------+-------------------+
  • Networks
incus network ls
+----------------+----------+-------+------+------+-------------+-------------+---------+
|      NOM       |   TYPE   | GÉRÉ  | IPV4 | IPV6 | DESCRIPTION | UTILISÉ PAR |  ÉTAT   |
+----------------+----------+-------+------+------+-------------+-------------+---------+
| UPLINK         | physical | true  |      |      |             | 0           | CREATED |
+----------------+----------+-------+------+------+-------------+-------------+---------+
| asw            | bridge   | false |      |      |             | 2           |         |
+----------------+----------+-------+------+------+-------------+-------------+---------+
| br-int         | bridge   | false |      |      |             | 0           |         |
+----------------+----------+-------+------+------+-------------+-------------+---------+
| eno1           | physical | false |      |      |             | 0           |         |
+----------------+----------+-------+------+------+-------------+-------------+---------+
| genev_sys_6081 | unknown  | false |      |      |             | 0           |         |
+----------------+----------+-------+------+------+-------------+-------------+---------+
| lo             | loopback | false |      |      |             | 0           |         |
+----------------+----------+-------+------+------+-------------+-------------+---------+
| ovs-system     | unknown  | false |      |      |             | 0           |         |
+----------------+----------+-------+------+------+-------------+-------------+---------+
| thunder0       | physical | false |      |      |             | 0           |         |
+----------------+----------+-------+------+------+-------------+-------------+---------+
| ttn0           | physical | false |      |      |             | 0           |         |
+----------------+----------+-------+------+------+-------------+-------------+---------+
| vlan2          | unknown  | false |      |      |             | 0           |         |
+----------------+----------+-------+------+------+-------------+-------------+---------+
| vlan3          | unknown  | false |      |      |             | 0           |         |
+----------------+----------+-------+------+------+-------------+-------------+---------+
| vlan4          | unknown  | false |      |      |             | 0           |         |
+----------------+----------+-------+------+------+-------------+-------------+---------+
| wlp0s20f3      | physical | false |      |      |             | 0           |         |
+----------------+----------+-------+------+------+-------------+-------------+---------+
  • Open vSwitch configuration
sudo ovs-vsctl show
175499ed-8c2a-4d03-92ce-4c06e32e26a7
    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port ovn-f13c85-0
            Interface ovn-f13c85-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="192.0.0.202"}
        Port ovn-97f0d5-0
            Interface ovn-97f0d5-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="192.0.0.201"}
        Port br-int
            Interface br-int
                type: internal
    Bridge asw
        Port vlan2
            tag: 2
            Interface vlan2
                type: internal
        Port vlan4
            tag: 4
            Interface vlan4
                type: internal
        Port thunder0
            Interface thunder0
        Port asw
            Interface asw
                type: internal
        Port vlan3
            tag: 3
            Interface vlan3
                type: internal
    ovs_version: "3.3.1"
  • OVN southbound → The chassis is there
sudo ovn-sbctl show
Chassis "97f0d589-c1e4-4040-a872-4f431d609f62"
    hostname: home1
    Encap geneve
        ip: "192.0.0.201"
        options: {csum="true"}
Chassis "0e79bfa6-7a96-4b59-bf1e-e80dadd44c8d"
    hostname: home0
    Encap geneve
        ip: "192.0.0.200"
        options: {csum="true"}
Chassis "f13c85d4-5088-483b-9160-344270beed8e"
    hostname: home2
    Encap geneve
        ip: "192.0.0.202"
        options: {csum="true"}
  • OVN northbound → empty. Is it the expected result?
sudo ovn-nbctl show

Did you try systemctl restart incus?

The OVN connection is attempted on startup or upon modification of the network.ovn.northbound_connection config key. So if OVN was brought online afterwards, that’d explain the error.

Thanks for your prompt reply.
I did many configuration attempts, and it seems the database is not in a good state.

  • I had forgotten the configuration keys.
incus config show
config:
  cluster.https_address: 192.0.0.200:9443
  core.https_address: 192.0.0.200:9443
  core.syslog_socket: "true"
  network.ovn.northbound_connection: 192.0.0.200
  • Restart logs:
sudo systemctl restart incus
time="2024-06-23T20:30:28+02:00" level=warning msg="Aborting heartbeat round" err="context canceled" local="192.0.0.200:9443" mode=normal
time="2024-06-23T20:30:28+02:00" level=warning msg="Rejecting heartbeat request as shutting down"
time="2024-06-23T20:30:28+02:00" level=error msg="Failed cleaning up operations"
time="2024-06-23T20:30:28+02:00" level=warning msg="Rejecting heartbeat request as shutting down"
time="2024-06-23T20:30:28+02:00" level=warning msg="Dqlite proxy failed" err="first: local -> remote: read unix @->@977c8: read: connection reset by peer" local="192.0.0.200:9443" name=dqlite remote="192.0.0.201:41430"
time="2024-06-23T20:30:28+02:00" level=error msg="Failed getting version during QEMU initialization: exit status 127"
time="2024-06-23T20:30:28+02:00" level=warning msg="Instance type not operational" driver=qemu err="Failed getting QEMU version" type=virtual-machine
time="2024-06-23T20:30:28+02:00" level=warning msg="Dqlite: attempt 1: server 192.0.0.200:9443: no known leader"
  • Network create attempt
incus network create my-ovn --type=ovn
Error: Failed creating pending network for member "home0": Network is not in pending state

Ah, okay, that northbound_connection string isn’t correct.

If you’re only dealing with a single machine deployment, just unset the key entirely.
If you’re dealing with an OVN cluster with a replicated database, it should instead look like

  network.ovn.northbound_connection: tcp:192.168.0.1:6641,tcp:192.168.0.2:6641,tcp:192.168.0.3:6641

Thanks a lot. That was it.

  • Keys:
incus config show
config:
  cluster.https_address: 192.0.0.200:9443
  core.https_address: 192.0.0.200:9443
  core.syslog_socket: "true"
  network.ovn.northbound_connection: tcp:192.0.0.200:6641,tcp:192.0.0.201:6641,tcp:192.0.0.202:6641
  • Networks:
incus network ls | grep true
| UPLINK         | physical | true  |                 |                           |             | 1           | CREATED |
| ovn-net        | ovn      | true  | 10.160.158.1/24 | fd42:1d43:6356:8008::1/64 |             | 0           | CREATED |