OVN Database validation error after Incus cluster update

Incus
,
1

Howdy,

Updated Incus cluster tonight and after the usual go around a few times getting all the nodes back online, all the ovn networks had disappeared from incus network ls

When trying to create new ovn networks or (re)set incus config set network.ovn.northbound_connection <ovn-northd-nb-db> the following error is displayed:

Error: failed to notify peer 192.168.1.11:8443: unable to connect to any endpoints: failed to connect to tcp:192.168.1.10:6641: database OVN_Northbound validation error (6): database model contains a model for table Static_MAC_Binding that does not exist in schema. database model contains a model for table Chassis_Template_Var that does not exist in schema. Mapper Error. Object type ovsmodel.NAT contains field GatewayPort (*string) ovs tag gateway_port: Column does not exist in schema. database model contains a model for table Mirror that does not exist in schema. Mapper Error. Object type ovsmodel.LogicalSwitchPort contains field MirrorRules ([]string) ovs tag mirror_rules: Column does not exist in schema. database model contains a client index for table Logical_Switch_Port that does not exist in schema. failed to connect to tcp:192.168.1.12:6641: database OVN_Northbound validation error (1): database model name () does not match schema (OVN_Northbound). failed to connect to tcp:192.168.1.11:6641: database OVN_Northbound validation error (1): database model name () does not match schema (OVN_Northbound)

# incus version
Client version: 6.15
Server version: 6.15

# ovn-nbctl --version
ovn-nbctl 22.03.3
Open vSwitch Library 2.17.9
DB Schema 6.1.0

If I’m unable to find a workaround I will probably resort to removing all the ovn networks from each instances and start the ovn setup again from scratch… however figured I would post here first!

Cheers,
Chris

2

Your OVN installation is too old for Incus. we have a baseline of OVN 23.03.0.

Your version is consistent with running Ubuntu 22.04. An Ubuntu 24.04 system would have a sufficient version. Alternatively I have some backported package in Stéphane Graber - OVN stable : Stéphane Graber though those will not be maintained for much longer.

3

Thanks @stgraber.

It is indeed 22.04. I’ll look at updating the OS.

Out of curiosity… is there a compatibility matrix published somewhere or do we just need to dig through the release notes?

4

That one should be listed in our requirements but apparently it isn’t, so I’ll be adding it shortly.

5
1 Like
6

Ok. I have updated to 24.04 and upgraded Incus.

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04.3 LTS
Release:        24.04
Codename:       noble

# incus version
Client version: 6.15
Server version: 6.15

# apt-cache policy incus
incus:
  Installed: 1:6.15-ubuntu24.04-202508170325
  Candidate: 1:6.15-ubuntu24.04-202508170325
  Version table:
 *** 1:6.15-ubuntu24.04-202508170325 500
        500 https://pkgs.zabbly.com/incus/stable noble/main amd64 Packages
        100 /var/lib/dpkg/status
     1:6.15-ubuntu24.04-202508140159 500
        500 https://pkgs.zabbly.com/incus/stable noble/main amd64 Packages
     1:6.0.5-ubuntu24.04-202508170336 500
        500 https://pkgs.zabbly.com/incus/lts-6.0 noble/main amd64 Packages
     1:6.0.4-ubuntu24.04-202506291720 500
        500 https://pkgs.zabbly.com/incus/lts-6.0 noble/main amd64 Packages
     6.0.0-1ubuntu0.3 500
        500 http://au.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages
        500 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Packages
     6.0.0-1 500
        500 http://au.archive.ubuntu.com/ubuntu noble/universe amd64 Package

OVN and OVS versions showing as updated:

# ovn-nbctl --version
ovn-nbctl 24.03.2
Open vSwitch Library 3.3.0
DB Schema 7.3.0

# ovn-sbctl --version
ovn-sbctl 24.03.2
Open vSwitch Library 3.3.0
DB Schema 20.33.0

# ovs-vsctl --version
ovs-vsctl (Open vSwitch) 3.3.0
DB Schema 8.5.0

However my actual clustered database still has the old schema which is giving the same error as OP.

# sudo ovsdb-client get-schema-version tcp:192.168.1.10:6641,tcp:192.168.1.11:6641,tcp:192.168.1.12:6641 OVN_Northbound
2025-08-24T06:35:37Z|00001|reconnect|INFO|tcp:192.168.1.10:6641: connecting...
2025-08-24T06:35:37Z|00002|reconnect|INFO|tcp:192.168.1.10:6641: connection attempt failed (Connection refused)
2025-08-24T06:35:37Z|00003|reconnect|INFO|tcp:192.168.1.11:6641: connecting...
2025-08-24T06:35:37Z|00004|reconnect|INFO|tcp:192.168.1.11:6641: connected
6.1.0

# sudo ovsdb-client get-schema-version tcp:192.168.1.10:6642,tcp:192.168.1.11:6642,tcp:192.168.1.12:6642 OVN_Southbound
2025-08-24T06:36:11Z|00001|reconnect|INFO|tcp:192.168.1.10:6642: connecting...
2025-08-24T06:36:11Z|00002|reconnect|INFO|tcp:192.168.1.10:6642: connection attempt failed (Connection refused)
2025-08-24T06:36:11Z|00003|reconnect|INFO|tcp:192.168.1.11:6642: connecting...
2025-08-24T06:36:11Z|00004|reconnect|INFO|tcp:192.168.1.11:6642: connected
20.21.0

I wont get into the headache it was updating from Jammy to Noble, one server in particular was a pain and probably not 100%, but it’s running.

Do I need to manually upgrade the db? Should have this been handled by Incus?

I just want to check before I start trying to handle it directly through OVN.

7

Manually updated both the nb and sb databases with ovsdb-client convert [SERVER] SCHEMA using the ovsschemas from ovn/ovn-nb.ovsschema at branch-24.03 · ovn-org/ovn · GitHub and ovn/ovn-sb.ovsschema at branch-24.03 · ovn-org/ovn · GitHub respectively.

For anyone else, double check you are on the right branch and using the correct schema versions.

# ovn-nbctl --version
ovn-nbctl 24.03.2
Open vSwitch Library 3.3.0
DB Schema 7.3.0

# ovn-sbctl --version
ovn-sbctl 24.03.2
Open vSwitch Library 3.3.0
DB Schema 20.33.0

And I can now see my OVN networks again :slight_smile: