IncusOS OVN Issues

Hello!

I’m trying to enable the OVN service on IncusOS.

Just as background, I am running these commands in Powershell

The first thing I noticed that incus admin os service show ovn returns a YAML file and incus admin os service edit ovn takes a JSON file.

So I used incus query /os/1.0/services/ovn > ovn.json, which gave me a JSON and created a new file.

I edited it to be:

{
	"config": {
		"database": "tcp:10.40.0.2:6642",
		"enabled": true,
		"ic_chassis": false,
		"tls_ca_certificate": "",
		"tls_client_certificate": "",
		"tls_client_key": "",
		"tunnel_address": "10.40.0.2",
		"tunnel_protocol": "geneve"
	},
	"state": {}
}

Reviewing the IncusOS OVN Documentation and the Incus OVN Documentation, I came up with the assumptions:

  1. database = external_ids:ovn-remote
  2. tunnel_address = external_ids:ovn-encap-ip
  3. tunnel_protocol = external_ids:ovn-encap-type

I get the output (new lines added for readability):

PS C:\Users\Chase\Downloads> Get-Content .\ovn.json | incus admin os service edit ovn
WARNING: The IncusOS API and configuration is subject to change

Error: Failed to run: 
ovs-vsctl set open_vswitch . 
external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 
external_ids:ovn-remote=tcp:10.40.0.2:6642 
external_ids:ovn-encap-type=geneve 
external_ids:ovn-encap-ip=10.40.0.2 
external_ids:ovn-is-interconn=false: exit status 1 (ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory))
PS C:\Users\Chase\Downloads>

I also noticed in the Incus OS Documentation, it has a different example with external_ids:ovn-remote=unix:/run/ovn/ovnsb_db.sock and external_ids:ovn-encap-ip=127.0.0.1. Using that example, I’d get:

{
	"config": {
		"database": "unix:/run/openvswitch/db.sock",
		"enabled": true,
		"ic_chassis": false,
		"tls_ca_certificate": "",
		"tls_client_certificate": "",
		"tls_client_key": "",
		"tunnel_address": "127.0.0.1",
		"tunnel_protocol": "geneve"
	},
	"state": {}
}

and

PS C:\Users\Chase\Downloads> Get-Content .\ovn.json | incus admin os service edit ovn
WARNING: The IncusOS API and configuration is subject to change

Error: Failed to run: 
ovs-vsctl set open_vswitch . 
external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 
external_ids:ovn-remote=unix:/run/openvswitch/db.sock 
external_ids:ovn-encap-type=geneve 
external_ids:ovn-encap-ip=127.0.0.1 
external_ids:ovn-is-interconn=false: exit status 1 (ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory))
PS C:\Users\Chase\Downloads>

Is there something I’m doing wrong?

IncusOS and OVN is something that I’ve implemented but then haven’t yet rolled out in the validation lab, so there’s quite a good chance that this is broken right now.

I’ve put a TODO list item for it for hopefully later today so I can try to reproduce the issue and get this working reliably in the validation lab.

1 Like

So I nuked the endpoint and reinstalled the OS, from scratch, and instead of using Windows Powershell, I used WSL with Debian.

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus remote add incus-02 10.40.0.2
Certificate fingerprint: [...]
ok (y/n/[fingerprint])? y
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus remote switch incus-02
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus network list
+----------+--------+---------+-----------------+---------------------------+----------------------------+---------+---------+
|   NAME   |  TYPE  | MANAGED |      IPV4       |           IPV6            |        DESCRIPTION         | USED BY |  STATE  |
+----------+--------+---------+-----------------+---------------------------+----------------------------+---------+---------+
| incusbr0 | bridge | YES     | 10.207.223.1/24 | fd42:fce8:7b36:1d77::1/64 | Local network bridge (NAT) | 1       | CREATED |
+----------+--------+---------+-----------------+---------------------------+----------------------------+---------+---------+
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus network edit incusbr0
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus network list
+----------+--------+---------+---------------+------+----------------------------+---------+---------+
|   NAME   |  TYPE  | MANAGED |     IPV4      | IPV6 |        DESCRIPTION         | USED BY |  STATE  |
+----------+--------+---------+---------------+------+----------------------------+---------+---------+
| incusbr0 | bridge | YES     | 10.100.0.2/24 | none | Local network bridge (NAT) | 1       | CREATED |
+----------+--------+---------+---------------+------+----------------------------+---------+---------+
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus admin os service show ovn
WARNING: The IncusOS API and configuration is subject to change

config:
  database: ""
  enabled: false
  ic_chassis: false
  tls_ca_certificate: ""
  tls_client_certificate: ""
  tls_client_key: ""
  tunnel_address: ""
  tunnel_protocol: ""
state: {}
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus admin os service edit ovn
WARNING: The IncusOS API and configuration is subject to change

Config parsing error: Failed to run: ovs-vsctl set open_vswitch . external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 external_ids:ovn-remote=tcp:10.40.0.2:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=10.40.0.2 external_ids:ovn-is-interconn=false: exit status 1 (ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory))
Press enter to open the editor again or ctrl+c to abort change
^C
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus version
Client version: 6.19.1
Server version: 6.19.1
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config trust add-certificate ~/incus-ui.crt
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.ca_crt

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.client_cert

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.client_key

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.integration_bridge

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.northbound_connection

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.network.ovs.connection

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$

I think it’s interesting that those configs aren’t returning anything when they’re present in the WebUI

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config set network.ovn.integration_bridge incusbr0
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config set network.ovn.northbound_connection unix:/run/ovn/ovnnb_db.sock
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config set network.ovn.network.ovs.connection unix:/run/openvswitch/db.sock
Error: cannot set 'network.ovn.network.ovs.connection' to 'unix:/run/openvswitch/db.sock': unknown key
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config set network.ovn.network.ovs.connection /run/openvswitch/db.sock
Error: cannot set 'network.ovn.network.ovs.connection' to '/run/openvswitch/db.sock': unknown key
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.integration_bridge
incusbr0
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.northbound_connection

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.network.ovs.connection

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus admin os service edit ovn
WARNING: The IncusOS API and configuration is subject to change

Config parsing error: Failed to run: ovs-vsctl set open_vswitch . external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 external_ids:ovn-remote=tcp:10.40.0.2:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=10.40.0.2 external_ids:ovn-is-interconn=false: exit status 1 (ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory))
Press enter to open the editor again or ctrl+c to abort change

Config parsing error: Failed to run: ovs-vsctl set open_vswitch . external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 external_ids:ovn-remote=tcp:10.40.0.2:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=127.0.0.1 external_ids:ovn-is-interconn=false: exit status 1 (ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory))
Press enter to open the editor again or ctrl+c to abort change

Config parsing error: Failed to run: ovs-vsctl set open_vswitch . external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 external_ids:ovn-remote=tcp:10.40.0.2:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=127.0.0.1 external_ids:ovn-is-interconn=false: exit status 1 (ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory))
Press enter to open the editor again or ctrl+c to abort change
^C
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus admin os service show ovn
WARNING: The IncusOS API and configuration is subject to change

config:
  database: tcp:10.40.0.2:6642
  enabled: true
  ic_chassis: false
  tls_ca_certificate: ""
  tls_client_certificate: ""
  tls_client_key: ""
  tunnel_address: 127.0.0.1
  tunnel_protocol: geneve
state: {}

When I poke the OVN service with the API:

chase@DESKTOP-2TUBIB8:~$ curl -k --cert curl-client.pem https://10.40.0.2:8443/os/1.0/services/ovn
{"type":"sync","status":"Success","status_code":200,"operation":"","error_code":0,"error":"","metadata":{"state":{},"config":{"enabled":true,"ic_chassis":false,"database":"tcp:10.40.0.2:6642","tls_client_certificate":"","tls_client_key":"","tls_ca_certificate":"","tunnel_address":"127.0.0.1","tunnel_protocol":"geneve"}}}
chase@DESKTOP-2TUBIB8:~$ curl -k --cert curl-client.pem https://10.40.0.2:8443/os/1.0/services/ovn/:reset
{"type":"error","status":"","status_code":0,"operation":"","error_code":501,"error":"not implemented","metadata":null}
chase@DESKTOP-2TUBIB8:~$

I get “not implemented”.

Heres the last couple entries for incus-osd.service:

chase@DESKTOP-2TUBIB8:~$ curl -k --cert curl-client.pem https://10.40.0.2:8443/os/1.0/debug/log | jq '[.metadata[] | select(._SYSTEMD_UNIT == "incus-osd.service")] | .[-5:]'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2093k    0 2093k    0     0  13.3M      0 --:--:-- --:--:-- --:--:-- 13.3M
[
  {
    "MESSAGE": "ovs|00002|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)",
    "PRIORITY": "3",
    "SYSLOG_FACILITY": "3",
    "SYSLOG_IDENTIFIER": "ovs-vsctl",
    "SYSLOG_TIMESTAMP": "Dec  3 23:39:16 ",
    "_BOOT_ID": "252c23c1ce714c118500cb920b5948b8",
    "_CAP_EFFECTIVE": "1ffffffffff",
    "_COMM": "ovs-vsctl",
    "_GID": "0",
    "_HOSTNAME": "f3bdd818-166e-11f0-996b-31eea3194a00",
    "_MACHINE_ID": "9ad4ea371459453aa524e32d60dc6567",
    "_PID": "2831",
    "_RUNTIME_SCOPE": "system",
    "_SELINUX_CONTEXT": "unconfined\n",
    "_SOURCE_REALTIME_TIMESTAMP": "1764805156303658",
    "_SYSTEMD_CGROUP": "/system.slice/incus-osd.service",
    "_SYSTEMD_INVOCATION_ID": "10d55b4fd1364c8d84a490c949f0b41a",
    "_SYSTEMD_SLICE": "system.slice",
    "_SYSTEMD_UNIT": "incus-osd.service",
    "_TRANSPORT": "syslog",
    "_UID": "0",
    "__CURSOR": "s=6ec19a00f3d046ffbba9c9f9d2afef56;i=94c;b=252c23c1ce714c118500cb920b5948b8;m=7213d766;t=64514b8708abd;x=e43c237a3be2eb28",
    "__MONOTONIC_TIMESTAMP": "1913902950",
    "__REALTIME_TIMESTAMP": "1764805156309693",
    "__SEQNUM": "2380",
    "__SEQNUM_ID": "6ec19a00f3d046ffbba9c9f9d2afef56"
  },
  {
    "MESSAGE": "ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 external_ids:ovn-remote=tcp:10.40.0.2:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=127.0.0.1 external_ids:ovn-is-interconn=false",
    "PRIORITY": "5",
    "SYSLOG_FACILITY": "3",
    "SYSLOG_IDENTIFIER": "ovs-vsctl",
    "SYSLOG_TIMESTAMP": "Dec  3 23:39:51 ",
    "_BOOT_ID": "252c23c1ce714c118500cb920b5948b8",
    "_CAP_EFFECTIVE": "1ffffffffff",
    "_COMM": "ovs-vsctl",
    "_EXE": "/usr/bin/ovs-vsctl",
    "_GID": "0",
    "_HOSTNAME": "f3bdd818-166e-11f0-996b-31eea3194a00",
    "_MACHINE_ID": "9ad4ea371459453aa524e32d60dc6567",
    "_PID": "2841",
    "_RUNTIME_SCOPE": "system",
    "_SELINUX_CONTEXT": "unconfined\n",
    "_SOURCE_REALTIME_TIMESTAMP": "1764805191885048",
    "_SYSTEMD_CGROUP": "/system.slice/incus-osd.service",
    "_SYSTEMD_INVOCATION_ID": "10d55b4fd1364c8d84a490c949f0b41a",
    "_SYSTEMD_SLICE": "system.slice",
    "_SYSTEMD_UNIT": "incus-osd.service",
    "_TRANSPORT": "syslog",
    "_UID": "0",
    "__CURSOR": "s=6ec19a00f3d046ffbba9c9f9d2afef56;i=94d;b=252c23c1ce714c118500cb920b5948b8;m=7432adcc;t=64514ba8f6122;x=f146e5892f167b78",
    "__MONOTONIC_TIMESTAMP": "1949478348",
    "__REALTIME_TIMESTAMP": "1764805191885090",
    "__SEQNUM": "2381",
    "__SEQNUM_ID": "6ec19a00f3d046ffbba9c9f9d2afef56"
  },
  {
    "MESSAGE": "ovs|00002|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)",
    "PRIORITY": "3",
    "SYSLOG_FACILITY": "3",
    "SYSLOG_IDENTIFIER": "ovs-vsctl",
    "SYSLOG_TIMESTAMP": "Dec  3 23:39:51 ",
    "_BOOT_ID": "252c23c1ce714c118500cb920b5948b8",
    "_CAP_EFFECTIVE": "1ffffffffff",
    "_COMM": "ovs-vsctl",
    "_EXE": "/usr/bin/ovs-vsctl",
    "_GID": "0",
    "_HOSTNAME": "f3bdd818-166e-11f0-996b-31eea3194a00",
    "_MACHINE_ID": "9ad4ea371459453aa524e32d60dc6567",
    "_PID": "2841",
    "_RUNTIME_SCOPE": "system",
    "_SELINUX_CONTEXT": "unconfined\n",
    "_SOURCE_REALTIME_TIMESTAMP": "1764805191885169",
    "_SYSTEMD_CGROUP": "/system.slice/incus-osd.service",
    "_SYSTEMD_INVOCATION_ID": "10d55b4fd1364c8d84a490c949f0b41a",
    "_SYSTEMD_SLICE": "system.slice",
    "_SYSTEMD_UNIT": "incus-osd.service",
    "_TRANSPORT": "syslog",
    "_UID": "0",
    "__CURSOR": "s=6ec19a00f3d046ffbba9c9f9d2afef56;i=94e;b=252c23c1ce714c118500cb920b5948b8;m=7432b017;t=64514ba8f636d;x=3f02b452385d9673",
    "__MONOTONIC_TIMESTAMP": "1949478935",
    "__REALTIME_TIMESTAMP": "1764805191885677",
    "__SEQNUM": "2382",
    "__SEQNUM_ID": "6ec19a00f3d046ffbba9c9f9d2afef56"
  },
  {
    "MESSAGE": "ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 external_ids:ovn-remote=tcp:10.40.0.2:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=127.0.0.1 external_ids:ovn-is-interconn=false",
    "PRIORITY": "5",
    "SYSLOG_FACILITY": "3",
    "SYSLOG_IDENTIFIER": "ovs-vsctl",
    "SYSLOG_TIMESTAMP": "Dec  3 23:40:02 ",
    "_BOOT_ID": "252c23c1ce714c118500cb920b5948b8",
    "_CAP_EFFECTIVE": "1ffffffffff",
    "_COMM": "ovs-vsctl",
    "_GID": "0",
    "_HOSTNAME": "f3bdd818-166e-11f0-996b-31eea3194a00",
    "_MACHINE_ID": "9ad4ea371459453aa524e32d60dc6567",
    "_PID": "2844",
    "_RUNTIME_SCOPE": "system",
    "_SELINUX_CONTEXT": "unconfined\n",
    "_SOURCE_REALTIME_TIMESTAMP": "1764805202789927",
    "_SYSTEMD_CGROUP": "/system.slice/incus-osd.service",
    "_SYSTEMD_INVOCATION_ID": "10d55b4fd1364c8d84a490c949f0b41a",
    "_SYSTEMD_SLICE": "system.slice",
    "_SYSTEMD_UNIT": "incus-osd.service",
    "_TRANSPORT": "syslog",
    "_UID": "0",
    "__CURSOR": "s=6ec19a00f3d046ffbba9c9f9d2afef56;i=94f;b=252c23c1ce714c118500cb920b5948b8;m=74d912ff;t=64514bb35c655;x=e1d97f5995b60dcc",
    "__MONOTONIC_TIMESTAMP": "1960383231",
    "__REALTIME_TIMESTAMP": "1764805202789973",
    "__SEQNUM": "2383",
    "__SEQNUM_ID": "6ec19a00f3d046ffbba9c9f9d2afef56"
  },
  {
    "MESSAGE": "ovs|00002|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)",
    "PRIORITY": "3",
    "SYSLOG_FACILITY": "3",
    "SYSLOG_IDENTIFIER": "ovs-vsctl",
    "SYSLOG_TIMESTAMP": "Dec  3 23:40:02 ",
    "_BOOT_ID": "252c23c1ce714c118500cb920b5948b8",
    "_CAP_EFFECTIVE": "1ffffffffff",
    "_COMM": "ovs-vsctl",
    "_GID": "0",
    "_HOSTNAME": "f3bdd818-166e-11f0-996b-31eea3194a00",
    "_MACHINE_ID": "9ad4ea371459453aa524e32d60dc6567",
    "_PID": "2844",
    "_RUNTIME_SCOPE": "system",
    "_SELINUX_CONTEXT": "unconfined\n",
    "_SOURCE_REALTIME_TIMESTAMP": "1764805202790040",
    "_SYSTEMD_CGROUP": "/system.slice/incus-osd.service",
    "_SYSTEMD_INVOCATION_ID": "10d55b4fd1364c8d84a490c949f0b41a",
    "_SYSTEMD_SLICE": "system.slice",
    "_SYSTEMD_UNIT": "incus-osd.service",
    "_TRANSPORT": "syslog",
    "_UID": "0",
    "__CURSOR": "s=6ec19a00f3d046ffbba9c9f9d2afef56;i=950;b=252c23c1ce714c118500cb920b5948b8;m=74d914bc;t=64514bb35c812;x=568dcb67bd4fc99d",
    "__MONOTONIC_TIMESTAMP": "1960383676",
    "__REALTIME_TIMESTAMP": "1764805202790418",
    "__SEQNUM": "2384",
    "__SEQNUM_ID": "6ec19a00f3d046ffbba9c9f9d2afef56"
  }
]
chase@DESKTOP-2TUBIB8:~$

I was hoping journalctl would have given me more information to work off of, hopefully it helps you though

Late night edit:
I also took a stab at just returning every unique “message”, where the JSON entry contained any of the four strings that I associate with OVN:

chase@DESKTOP-2TUBIB8:~$ curl -k --cert curl-client.pem https://10.40.0.2:8443/os/1.0/debug/log | jq '[.metadata[] | select(any(.[]; tostring | test("ovn|ovs|vswitch|vsctl"; "i")))]  | .[].MESSAGE' | sort | uniq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2093k    0 2093k    0     0  14.0M      0 --:--:-- --:--:-- --:--:-- 14.0M
"ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 external_ids:ovn-remote=tcp:10.40.0.2:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=10.40.0.2 external_ids:ovn-is-interconn=false"
"ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 external_ids:ovn-remote=tcp:10.40.0.2:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=127.0.0.1 external_ids:ovn-is-interconn=false"
"ovs|00002|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)"

Nothing too groundbreaking.

I was looking at the repo and in incus-osd/internal/services/service_ovn.go. I don’t know GO, I’m more familiar with other languages, but here’s what I’ve pieced together:

On line… (sigh)… 67, if the service isn’t enabled and the new config sets it to enable, it’ll run Start(ctx). Inside of Start(), at line 105, it enables the service for ovs-vswitch: systemd.StartUnit(ctx, "ovs-vswitchd.service").

The error I got is usually related to the ovsdb-server service not running.

If we reference line 90, which is run when OVN is disabled, it also stops a few other ovn-related services: systemd.StopUnit(ctx, "ovn-controller.service", "ovs-vswitchd.service", "ovsdb-server.service")

I was considering cloning the repo and testing my guess before commenting about it, but I haven’t had a chance yet today and I probably wont for another 8 hours.

My initial assumption is to change this:

	// Start OVS.
	err := systemd.StartUnit(ctx, "ovs-vswitchd.service")
	if err != nil {
		return err
	}

into this:

	// Start OVS.
	err := systemd.StartUnit(ctx, "ovn-controller.service", "ovs-vswitchd.service", "ovsdb-server.service")
	if err != nil {
		return err
	}

If I understand the codebase properly, thatll start the other OVN/OVS services that IncusOS is expecting to run. After OVS is started, line 101 enters the configure() function, and then line 135 (_, err := subprocess.RunCommand("ovs-vsctl", args...)) is what generates the error for the socket missing (presumably because ovsdb-server wasn’t running by this point.

Looking into this issue now. I’ve got it reproduced here.

Awesome, I havent had very much time to look at this. Ive looked through the source a few more times and im a bit more confident that my initial thought is probably part of the issue but i havent been able build the source and test it.

Latest testing build works here:

stgraber@castiana:~ (incus:dev-cluster/default)$ incus list
+------+---------+----------------------+-------------------------------------------------+-----------------+-----------+----------+
| NAME |  STATE  |         IPV4         |                      IPV6                       |      TYPE       | SNAPSHOTS | LOCATION |
+------+---------+----------------------+-------------------------------------------------+-----------------+-----------+----------+
| d13  | RUNNING | 10.20.162.2 (enp5s0) | fd42:e9ac:c3e5:65f:1266:6aff:fe98:50ac (enp5s0) | VIRTUAL-MACHINE | 0         | incus03  |
+------+---------+----------------------+-------------------------------------------------+-----------------+-----------+----------+
stgraber@castiana:~ (incus:dev-cluster/default)$ incus list --project internal
+--------------------+---------+------+------------------------------------------------+-----------+-----------+----------+
|        NAME        |  STATE  | IPV4 |                      IPV6                      |   TYPE    | SNAPSHOTS | LOCATION |
+--------------------+---------+------+------------------------------------------------+-----------+-----------+----------+
| linstor-controller | RUNNING |      | fd42:143d:672d:ff60:1266:6aff:fee5:7eb1 (eth0) | CONTAINER | 0         | incus01  |
+--------------------+---------+------+------------------------------------------------+-----------+-----------+----------+
| ovn-central        | RUNNING |      | fd42:143d:672d:ff60:1266:6aff:fe6c:e5ac (eth0) | CONTAINER | 0         | incus02  |
+--------------------+---------+------+------------------------------------------------+-----------+-----------+----------+
stgraber@castiana:~ (incus:dev-cluster/default)$ incus network list
+----------+--------+---------+-----------------+---------------------------+------------------------------+---------+---------+
|   NAME   |  TYPE  | MANAGED |      IPV4       |           IPV6            |         DESCRIPTION          | USED BY |  STATE  |
+----------+--------+---------+-----------------+---------------------------+------------------------------+---------+---------+
| default  | ovn    | YES     | 10.20.162.1/24  | fd42:e9ac:c3e5:65f::1/64  |                              | 1       | CREATED |
+----------+--------+---------+-----------------+---------------------------+------------------------------+---------+---------+
| incusbr0 | bridge | YES     | 10.167.140.1/24 | fd42:c9f0:6f69:913f::1/64 | Local network bridge (NAT)   | 2       | CREATED |
+----------+--------+---------+-----------------+---------------------------+------------------------------+---------+---------+
| meshbr0  | bridge | YES     | none            | fd42:143d:672d:ff60::/64  | Internal mesh network bridge | 3       | CREATED |
+----------+--------+---------+-----------------+---------------------------+------------------------------+---------+---------+
stgraber@castiana:~ (incus:dev-cluster/default)$ incus config show
config:
  cluster.https_address: '[2602:fc62:b:8007:1266:6aff:fe46:8bd]:8443'
  core.bgp_address: :149
  core.bgp_asn: "65001"
  core.bgp_routerid: 10.0.0.101
  core.https_address: :8443
  network.ovn.northbound_connection: tcp:[fd42:143d:672d:ff60:1266:6aff:fe6c:e5ac]:6641
  storage.backups_volume: local/backups
  storage.images_volume: local/images
  storage.linstor.controller_connection: http://[fd42:143d:672d:ff60:1266:6aff:fee5:7eb1]:3370
stgraber@castiana:~ (incus:dev-cluster/default)$ incus cluster list
+---------+-----------------------------------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
|  NAME   |                         URL                         |      ROLES      | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATUS |      MESSAGE      |
+---------+-----------------------------------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| incus01 | https://[2602:fc62:b:8007:1266:6aff:fe46:8bd]:8443  | database        | x86_64       | default        |             | ONLINE | Fully operational |
+---------+-----------------------------------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| incus02 | https://[2602:fc62:b:8007:1266:6aff:fe02:fa52]:8443 | database-leader | x86_64       | default        |             | ONLINE | Fully operational |
|         |                                                     | database        |              |                |             |        |                   |
+---------+-----------------------------------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| incus03 | https://[2602:fc62:b:8007:1266:6aff:fed2:70e9]:8443 | database        | x86_64       | default        |             | ONLINE | Fully operational |
+---------+-----------------------------------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
stgraber@castiana:~ (incus:dev-cluster/default)$ 

I need to do more manual tests of other aspects of that image before we push it to stable, but at least on the OVN front, it’s looking good.

Bringing this back up, I tried to enable OVN by editing incus admin os service show ovn and I wasn’t able to quite figure it out.

When reviewing https://linuxcontainers.org/incus-os/docs/main/reference/services/ovn/, it didn’t give me too much guidance, so I tried just setting the config to enabled, then the tunnel_address and tunnel_protocol, and eventually, it seemed to accept the config after I added the database string with all three IPs. But I was never able to get it to update the state.

I had this config on all three nodes:

config:
  database: tcp:10.40.0.2:6642,tcp:10.40.0.3:6642,tcp:10.40.0.4:6642
  enabled: true
  tunnel_address: 10.40.0.2
  tunnel_protocol: geneve
state: {}

(Other two nodes had their own IP in tunnel_address)

After I set that config on all three nodes, I tried to set network.ovn.northbound_connection: tcp:10.40.0.2:6642,tcp:10.40.0.3:6642,tcp:10.40.0.4:6642 inside of incus config edit, which didn’t take either.

Is there something I’m missing?

Here is a dump of the logs from 10.40.0.2. I assume 10.40.0.3 and 10.40.0.4 would be similar.

[chase@chase-laptop ~]$ incus admin os debug log | grep ovn
WARNING: The IncusOS API and configuration is subject to change

[2026/01/25 13:33:34 CST] ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=b5bfc99c-166c-11f0-bd1c-e8d4db9f5700 external_ids:ovn-remote= external_ids:ovn-encap-type= external_ids:ovn-encap-ip= external_ids:ovn-is-interconn=false
[2026/01/25 13:33:34 CST] ovs-vsctl: ovs|00002|db_ctl_base|ERR|external_ids:ovn-remote=: argument does not end in "=" followed by a value.
[2026/01/25 13:34:37 CST] ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=b5bfc99c-166c-11f0-bd1c-e8d4db9f5700 external_ids:ovn-remote= external_ids:ovn-encap-type= external_ids:ovn-encap-ip= external_ids:ovn-is-interconn=false
[2026/01/25 13:34:37 CST] ovs-vsctl: ovs|00002|db_ctl_base|ERR|external_ids:ovn-remote=: argument does not end in "=" followed by a value.
[2026/01/25 13:39:11 CST] ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=b5bfc99c-166c-11f0-bd1c-e8d4db9f5700 external_ids:ovn-remote= external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=10.40.0.2 external_ids:ovn-is-interconn=false
[2026/01/25 13:39:11 CST] ovs-vsctl: ovs|00002|db_ctl_base|ERR|external_ids:ovn-remote=: argument does not end in "=" followed by a value.
[2026/01/25 13:39:51 CST] ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=b5bfc99c-166c-11f0-bd1c-e8d4db9f5700 external_ids:ovn-remote= external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=10.40.0.2 external_ids:ovn-is-interconn=false
[2026/01/25 13:39:51 CST] ovs-vsctl: ovs|00002|db_ctl_base|ERR|external_ids:ovn-remote=: argument does not end in "=" followed by a value.
[2026/01/25 13:42:06 CST] ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=b5bfc99c-166c-11f0-bd1c-e8d4db9f5700 external_ids:ovn-remote=tcp:10.40.0.2:6642,tcp:10.40.0.3:6642,tcp:10.40.0.4:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=10.40.0.2 external_ids:ovn-is-interconn=false
[2026/01/25 13:42:06 CST] systemd: Configuration file /run/systemd/system/ovn-controller.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[2026/01/25 13:42:06 CST] systemd: Started ovn-controller.service - Open Virtual Network host control daemon.
[2026/01/25 13:42:06 CST] kernel: audit: type=1130 audit(1769370126.873:549): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=ovn-controller comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00001|reconnect|INFO|unix:/run/openvswitch/db.sock: connecting...
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00002|reconnect|INFO|unix:/run/openvswitch/db.sock: connected
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00001|reconnect|INFO|unix:/run/openvswitch/db.sock: connecting...
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00002|reconnect|INFO|unix:/run/openvswitch/db.sock: connected
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00003|main|INFO|OVN internal version is : [25.09.2-21.5.0-81.10]
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00003|main|INFO|OVN internal version is : [25.09.2-21.5.0-81.10]
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00004|main|INFO|OVS IDL reconnected, force recompute.
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00004|main|INFO|OVS IDL reconnected, force recompute.
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00005|reconnect|INFO|tcp:10.40.0.4:6642: connecting...
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00005|reconnect|INFO|tcp:10.40.0.4:6642: connecting...
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00006|main|INFO|OVNSB IDL reconnected, force recompute.
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00006|main|INFO|OVNSB IDL reconnected, force recompute.
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00007|main|INFO|Setting flow table prefixes: ip_src, ip_dst, ipv6_src, ipv6_dst.
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00008|ovn_util|INFO|statctrl: connecting to switch: "unix://var/run/openvswitch/br-int.mgmt"
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00007|main|INFO|Setting flow table prefixes: ip_src, ip_dst, ipv6_src, ipv6_dst.
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00009|rconn|INFO|unix://var/run/openvswitch/br-int.mgmt: connecting...
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00010|rconn|WARN|unix://var/run/openvswitch/br-int.mgmt: connection failed (No such file or directory)
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00011|rconn|INFO|unix://var/run/openvswitch/br-int.mgmt: waiting 1 seconds before reconnect
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00012|ovn_util|INFO|pinctrl: connecting to switch: "unix://var/run/openvswitch/br-int.mgmt"
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00013|rconn|INFO|unix://var/run/openvswitch/br-int.mgmt: connecting...
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00008|ovn_util|INFO|statctrl: connecting to switch: "unix://var/run/openvswitch/br-int.mgmt"
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00014|rconn|WARN|unix://var/run/openvswitch/br-int.mgmt: connection failed (No such file or directory)
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00015|rconn|INFO|unix://var/run/openvswitch/br-int.mgmt: waiting 1 seconds before reconnect
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00009|rconn|INFO|unix://var/run/openvswitch/br-int.mgmt: connecting...
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00010|rconn|WARN|unix://var/run/openvswitch/br-int.mgmt: connection failed (No such file or directory)
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00011|rconn|INFO|unix://var/run/openvswitch/br-int.mgmt: waiting 1 seconds before reconnect
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00012|ovn_util|INFO|pinctrl: connecting to switch: "unix://var/run/openvswitch/br-int.mgmt"
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00013|rconn|INFO|unix://var/run/openvswitch/br-int.mgmt: connecting...
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00014|rconn|WARN|unix://var/run/openvswitch/br-int.mgmt: connection failed (No such file or directory)
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00015|rconn|INFO|unix://var/run/openvswitch/br-int.mgmt: waiting 1 seconds before reconnect
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00016|reconnect|INFO|tcp:10.40.0.4:6642: connection attempt failed (Connection refused)
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00016|reconnect|INFO|tcp:10.40.0.4:6642: connection attempt failed (Connection refused)
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00017|reconnect|INFO|tcp:10.40.0.2:6642: connecting...
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00018|reconnect|INFO|tcp:10.40.0.2:6642: connection attempt failed (Connection refused)
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00017|reconnect|INFO|tcp:10.40.0.2:6642: connecting...
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00019|reconnect|INFO|tcp:10.40.0.3:6642: connecting...
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00018|reconnect|INFO|tcp:10.40.0.2:6642: connection attempt failed (Connection refused)
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00019|reconnect|INFO|tcp:10.40.0.3:6642: connecting...
[2026/01/25 13:42:06 CST] ovn-controller: 2026-01-25T19:42:06Z|00020|reconnect|INFO|tcp:10.40.0.3:6642: connection attempt failed (Connection refused)
[2026/01/25 13:42:06 CST] ovn-controller: ovs|00020|reconnect|INFO|tcp:10.40.0.3:6642: connection attempt failed (Connection refused)
[2026/01/25 13:42:07 CST] ovn-controller: 2026-01-25T19:42:07Z|00001|rconn(ovn_pinctrl0)|INFO|unix://var/run/openvswitch/br-int.mgmt: connecting...
[2026/01/25 13:42:07 CST] ovn-controller: 2026-01-25T19:42:07Z|00001|rconn(ovn_statctrl3)|INFO|unix://var/run/openvswitch/br-int.mgmt: connecting...
[2026/01/25 13:42:07 CST] ovn-controller: ovs|00001|rconn(ovn_pinctrl0)|INFO|unix://var/run/openvswitch/br-int.mgmt: connecting...
[2026/01/25 13:42:07 CST] ovn-controller: 2026-01-25T19:42:07Z|00021|reconnect|INFO|tcp:10.40.0.4:6642: connecting...
[2026/01/25 13:42:07 CST] ovn-controller: ovs|00001|rconn(ovn_statctrl3)|INFO|unix://var/run/openvswitch/br-int.mgmt: connecting...
[2026/01/25 13:42:07 CST] ovn-controller: ovs|00021|reconnect|INFO|tcp:10.40.0.4:6642: connecting...
[2026/01/25 13:42:07 CST] ovn-controller: 2026-01-25T19:42:07Z|00022|reconnect|INFO|tcp:10.40.0.4:6642: connection attempt failed (Connection refused)
[2026/01/25 13:42:07 CST] ovn-controller: 2026-01-25T19:42:07Z|00023|reconnect|INFO|tcp:10.40.0.4:6642: waiting 2 seconds before reconnect
[2026/01/25 13:42:07 CST] ovn-controller: ovs|00022|reconnect|INFO|tcp:10.40.0.4:6642: connection attempt failed (Connection refused)
[2026/01/25 13:42:07 CST] ovn-controller: ovs|00023|reconnect|INFO|tcp:10.40.0.4:6642: waiting 2 seconds before reconnect
[2026/01/25 13:42:07 CST] ovn-controller: 2026-01-25T19:42:07Z|00002|rconn(ovn_pinctrl0)|INFO|unix://var/run/openvswitch/br-int.mgmt: connected
[2026/01/25 13:42:07 CST] ovn-controller: ovs|00002|rconn(ovn_pinctrl0)|INFO|unix://var/run/openvswitch/br-int.mgmt: connected
[2026/01/25 13:42:07 CST] ovn-controller: 2026-01-25T19:42:07Z|00002|rconn(ovn_statctrl3)|INFO|unix://var/run/openvswitch/br-int.mgmt: connected
[2026/01/25 13:42:07 CST] ovn-controller: ovs|00002|rconn(ovn_statctrl3)|INFO|unix://var/run/openvswitch/br-int.mgmt: connected
[2026/01/25 13:42:09 CST] ovn-controller: 2026-01-25T19:42:09Z|00024|reconnect|INFO|tcp:10.40.0.2:6642: connecting...
[2026/01/25 13:42:09 CST] ovn-controller: ovs|00024|reconnect|INFO|tcp:10.40.0.2:6642: connecting...
[2026/01/25 13:42:09 CST] ovn-controller: 2026-01-25T19:42:09Z|00025|reconnect|INFO|tcp:10.40.0.2:6642: connection attempt failed (Connection refused)
[2026/01/25 13:42:09 CST] ovn-controller: 2026-01-25T19:42:09Z|00026|reconnect|INFO|tcp:10.40.0.2:6642: waiting 4 seconds before reconnect
[2026/01/25 13:42:09 CST] ovn-controller: ovs|00025|reconnect|INFO|tcp:10.40.0.2:6642: connection attempt failed (Connection refused)
[2026/01/25 13:42:09 CST] ovn-controller: ovs|00026|reconnect|INFO|tcp:10.40.0.2:6642: waiting 4 seconds before reconnect
[2026/01/25 13:42:13 CST] ovn-controller: 2026-01-25T19:42:13Z|00027|reconnect|INFO|tcp:10.40.0.3:6642: connecting...
[2026/01/25 13:42:13 CST] ovn-controller: ovs|00027|reconnect|INFO|tcp:10.40.0.3:6642: connecting...
[2026/01/25 13:42:13 CST] ovn-controller: 2026-01-25T19:42:13Z|00028|reconnect|INFO|tcp:10.40.0.3:6642: connection attempt failed (Connection refused)
[2026/01/25 13:42:13 CST] ovn-controller: 2026-01-25T19:42:13Z|00029|reconnect|INFO|tcp:10.40.0.3:6642: continuing to reconnect in the background but suppressing further logging
[2026/01/25 13:42:13 CST] ovn-controller: ovs|00028|reconnect|INFO|tcp:10.40.0.3:6642: connection attempt failed (Connection refused)
[2026/01/25 13:42:13 CST] ovn-controller: ovs|00029|reconnect|INFO|tcp:10.40.0.3:6642: continuing to reconnect in the background but suppressing further logging
[2026/01/25 13:42:21 CST] ovn-controller: 2026-01-25T19:42:21Z|00030|memory|INFO|14476 kB peak resident set size after 15.0 seconds
[2026/01/25 13:42:21 CST] ovn-controller: 2026-01-25T19:42:21Z|00031|memory|INFO|idl-cells-Open_vSwitch:107
[2026/01/25 13:42:21 CST] ovn-controller: ovs|00030|memory|INFO|14476 kB peak resident set size after 15.0 seconds
[2026/01/25 13:42:21 CST] ovn-controller: ovs|00031|memory|INFO|idl-cells-Open_vSwitch:107
[2026/01/25 14:12:37 CST] ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=b5bfc99c-166c-11f0-bd1c-e8d4db9f5700 external_ids:ovn-remote= external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=10.40.0.2 external_ids:ovn-is-interconn=false
[2026/01/25 14:12:37 CST] ovs-vsctl: ovs|00002|db_ctl_base|ERR|external_ids:ovn-remote=: argument does not end in "=" followed by a value.
[2026/01/25 15:21:25 CST] ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=b5bfc99c-166c-11f0-bd1c-e8d4db9f5700 external_ids:ovn-remote=tcp:10.40.0.2:6642,tcp:10.40.0.3:6642,tcp:10.40.0.4:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=10.40.0.2 external_ids:ovn-is-interconn=false
[2026/01/25 15:21:25 CST] systemd: Configuration file /run/systemd/system/ovn-controller.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00032|fatal_signal|WARN|terminating with signal 15 (Terminated)
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00032|fatal_signal|WARN|terminating with signal 15 (Terminated)
[2026/01/25 15:21:25 CST] systemd: Stopping ovn-controller.service - Open Virtual Network host control daemon...
[2026/01/25 15:21:25 CST] systemd: ovn-controller.service: Deactivated successfully.
[2026/01/25 15:21:25 CST] systemd: Stopped ovn-controller.service - Open Virtual Network host control daemon.
[2026/01/25 15:21:25 CST] kernel: audit: type=1131 audit(1769376085.753:576): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=ovn-controller comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2026/01/25 15:21:25 CST] systemd: Started ovn-controller.service - Open Virtual Network host control daemon.
[2026/01/25 15:21:25 CST] kernel: audit: type=1130 audit(1769376085.755:577): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=ovn-controller comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00001|reconnect|INFO|unix:/run/openvswitch/db.sock: connecting...
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00001|reconnect|INFO|unix:/run/openvswitch/db.sock: connecting...
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00002|reconnect|INFO|unix:/run/openvswitch/db.sock: connected
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00002|reconnect|INFO|unix:/run/openvswitch/db.sock: connected
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00003|main|INFO|OVN internal version is : [25.09.2-21.5.0-81.10]
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00003|main|INFO|OVN internal version is : [25.09.2-21.5.0-81.10]
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00004|main|INFO|OVS IDL reconnected, force recompute.
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00005|reconnect|INFO|tcp:10.40.0.3:6642: connecting...
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00006|main|INFO|OVNSB IDL reconnected, force recompute.
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00004|main|INFO|OVS IDL reconnected, force recompute.
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00007|ovn_util|INFO|statctrl: connecting to switch: "unix://var/run/openvswitch/br-int.mgmt"
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00008|rconn|INFO|unix://var/run/openvswitch/br-int.mgmt: connecting...
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00009|ovn_util|INFO|pinctrl: connecting to switch: "unix://var/run/openvswitch/br-int.mgmt"
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00010|rconn|INFO|unix://var/run/openvswitch/br-int.mgmt: connecting...
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00005|reconnect|INFO|tcp:10.40.0.3:6642: connecting...
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00006|main|INFO|OVNSB IDL reconnected, force recompute.
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00001|rconn(ovn_statctrl3)|INFO|unix://var/run/openvswitch/br-int.mgmt: connected
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00001|rconn(ovn_pinctrl0)|INFO|unix://var/run/openvswitch/br-int.mgmt: connected
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00007|ovn_util|INFO|statctrl: connecting to switch: "unix://var/run/openvswitch/br-int.mgmt"
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00011|reconnect|INFO|tcp:10.40.0.3:6642: connection attempt failed (Connection refused)
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00008|rconn|INFO|unix://var/run/openvswitch/br-int.mgmt: connecting...
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00012|reconnect|INFO|tcp:10.40.0.2:6642: connecting...
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00013|reconnect|INFO|tcp:10.40.0.2:6642: connection attempt failed (Connection refused)
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00009|ovn_util|INFO|pinctrl: connecting to switch: "unix://var/run/openvswitch/br-int.mgmt"
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00014|reconnect|INFO|tcp:10.40.0.4:6642: connecting...
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00010|rconn|INFO|unix://var/run/openvswitch/br-int.mgmt: connecting...
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00001|rconn(ovn_statctrl3)|INFO|unix://var/run/openvswitch/br-int.mgmt: connected
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00001|rconn(ovn_pinctrl0)|INFO|unix://var/run/openvswitch/br-int.mgmt: connected
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00011|reconnect|INFO|tcp:10.40.0.3:6642: connection attempt failed (Connection refused)
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00012|reconnect|INFO|tcp:10.40.0.2:6642: connecting...
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00013|reconnect|INFO|tcp:10.40.0.2:6642: connection attempt failed (Connection refused)
[2026/01/25 15:21:25 CST] ovn-controller: 2026-01-25T21:21:25Z|00015|reconnect|INFO|tcp:10.40.0.4:6642: connection attempt failed (Connection refused)
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00014|reconnect|INFO|tcp:10.40.0.4:6642: connecting...
[2026/01/25 15:21:25 CST] ovn-controller: ovs|00015|reconnect|INFO|tcp:10.40.0.4:6642: connection attempt failed (Connection refused)
[2026/01/25 15:21:26 CST] ovn-controller: 2026-01-25T21:21:26Z|00016|reconnect|INFO|tcp:10.40.0.3:6642: connecting...
[2026/01/25 15:21:26 CST] ovn-controller: ovs|00016|reconnect|INFO|tcp:10.40.0.3:6642: connecting...
[2026/01/25 15:21:26 CST] ovn-controller: 2026-01-25T21:21:26Z|00017|reconnect|INFO|tcp:10.40.0.3:6642: connection attempt failed (Connection refused)
[2026/01/25 15:21:26 CST] ovn-controller: 2026-01-25T21:21:26Z|00018|reconnect|INFO|tcp:10.40.0.3:6642: waiting 2 seconds before reconnect
[2026/01/25 15:21:26 CST] ovn-controller: ovs|00017|reconnect|INFO|tcp:10.40.0.3:6642: connection attempt failed (Connection refused)
[2026/01/25 15:21:26 CST] ovn-controller: ovs|00018|reconnect|INFO|tcp:10.40.0.3:6642: waiting 2 seconds before reconnect
[2026/01/25 15:21:28 CST] ovn-controller: 2026-01-25T21:21:28Z|00019|reconnect|INFO|tcp:10.40.0.2:6642: connecting...
[2026/01/25 15:21:28 CST] ovn-controller: ovs|00019|reconnect|INFO|tcp:10.40.0.2:6642: connecting...
[2026/01/25 15:21:28 CST] ovn-controller: 2026-01-25T21:21:28Z|00020|reconnect|INFO|tcp:10.40.0.2:6642: connection attempt failed (Connection refused)
[2026/01/25 15:21:28 CST] ovn-controller: 2026-01-25T21:21:28Z|00021|reconnect|INFO|tcp:10.40.0.2:6642: waiting 4 seconds before reconnect
[2026/01/25 15:21:28 CST] ovn-controller: ovs|00020|reconnect|INFO|tcp:10.40.0.2:6642: connection attempt failed (Connection refused)
[2026/01/25 15:21:28 CST] ovn-controller: ovs|00021|reconnect|INFO|tcp:10.40.0.2:6642: waiting 4 seconds before reconnect
[2026/01/25 15:21:32 CST] ovn-controller: 2026-01-25T21:21:32Z|00022|reconnect|INFO|tcp:10.40.0.4:6642: connecting...
[2026/01/25 15:21:32 CST] ovn-controller: ovs|00022|reconnect|INFO|tcp:10.40.0.4:6642: connecting...
[2026/01/25 15:21:32 CST] ovn-controller: 2026-01-25T21:21:32Z|00023|reconnect|INFO|tcp:10.40.0.4:6642: connection attempt failed (Connection refused)
[2026/01/25 15:21:32 CST] ovn-controller: 2026-01-25T21:21:32Z|00024|reconnect|INFO|tcp:10.40.0.4:6642: continuing to reconnect in the background but suppressing further logging
[2026/01/25 15:21:32 CST] ovn-controller: ovs|00023|reconnect|INFO|tcp:10.40.0.4:6642: connection attempt failed (Connection refused)
[2026/01/25 15:21:32 CST] ovn-controller: ovs|00024|reconnect|INFO|tcp:10.40.0.4:6642: continuing to reconnect in the background but suppressing further logging
[2026/01/25 15:21:40 CST] ovn-controller: 2026-01-25T21:21:40Z|00025|memory|INFO|14476 kB peak resident set size after 15.0 seconds
[2026/01/25 15:21:40 CST] ovn-controller: 2026-01-25T21:21:40Z|00026|memory|INFO|idl-cells-Open_vSwitch:107
[2026/01/25 15:21:40 CST] ovn-controller: ovs|00025|memory|INFO|14476 kB peak resident set size after 15.0 seconds
[2026/01/25 15:21:40 CST] ovn-controller: ovs|00026|memory|INFO|idl-cells-Open_vSwitch:107
[2026/01/25 15:23:42 CST] ovn-controller: 2026-01-25T21:23:42Z|00002|fatal_signal(ovn_statctrl3)|WARN|terminating with signal 15 (Terminated)
[2026/01/25 15:23:42 CST] kernel: audit: type=1131 audit(1769376222.114:600): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=ovn-controller comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2026/01/25 15:23:42 CST] ovn-controller: ovs|00002|fatal_signal(ovn_statctrl3)|WARN|terminating with signal 15 (Terminated)
[2026/01/25 15:23:42 CST] systemd: Stopping ovn-controller.service - Open Virtual Network host control daemon...
[2026/01/25 15:23:42 CST] systemd: ovn-controller.service: Deactivated successfully.
[2026/01/25 15:23:42 CST] systemd: Stopped ovn-controller.service - Open Virtual Network host control daemon.
[2026/01/25 15:23:42 CST] systemd: Configuration file /run/systemd/system/ovn-controller.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[2026/01/25 15:23:42 CST] systemd: Configuration file /run/systemd/system/ovn-controller.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[2026/01/25 15:23:42 CST] ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=b5bfc99c-166c-11f0-bd1c-e8d4db9f5700 external_ids:ovn-remote= external_ids:ovn-encap-type= external_ids:ovn-encap-ip= external_ids:ovn-is-interconn=false
[2026/01/25 15:23:57 CST] ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=b5bfc99c-166c-11f0-bd1c-e8d4db9f5700 external_ids:ovn-remote= external_ids:ovn-encap-type= external_ids:ovn-encap-ip= external_ids:ovn-is-interconn=false
[2026/01/25 15:24:15 CST] ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=b5bfc99c-166c-11f0-bd1c-e8d4db9f5700 external_ids:ovn-remote= external_ids:ovn-encap-type= external_ids:ovn-encap-ip= external_ids:ovn-is-interconn=false
[2026/01/25 15:24:17 CST] ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=b5bfc99c-166c-11f0-bd1c-e8d4db9f5700 external_ids:ovn-remote= external_ids:ovn-encap-type= external_ids:ovn-encap-ip= external_ids:ovn-is-interconn=false
[chase@chase-laptop ~]$ 
[chase@chase-laptop ~]$ incus admin os show 
WARNING: The IncusOS API and configuration is subject to change

environment:
  hostname: b5bfc99c-166c-11f0-bd1c-e8d4db9f5700
  os_name: IncusOS
  os_version: "202601220238"
  os_version_next: ""
[chase@chase-laptop ~]$ 

I think the problem is that you haven’t actually deployed the OVN control plane.

Your DB string shows 3 IP addresses but then one of those at least is an IncusOS server. IncusOS ships the client bits of OVN (ovn-controller), it does not ship any of the control plane bits (ovn-central).

You need to setup ovn-central somewhere, either with just a standalone instance of it or a cluster of it (for HA), then you can have your IncusOS servers point to that ovn-central deployment.

In my own environments, I usually deploy ovn-central in HA with 3 containers running on the same Incus cluster that uses that ovn-central, just making sure not to use OVN for those containers as that’d obviously lead to failure :slight_smile:

Ah that’ll do it. I guess I didn’t realize IncusOS doesn’t manage the client- and server-side of OVN since that’s bundled together in the ‘normal’ Incus docs lol

So I’ve spent some time looking at that and getting a little more familiar with how virtual networking works. When you do this, do you set up a physical bridge and have the container with ovn-central pull an IP directly from the router or do you set it up on the default IncusOS NAT’d bridge and configure the hosts to forward traffic to those CTs?

I guess where it stands right now:

I have the default Linux bridge that’s created in IncusOS (incusbr0), which is my management control plane and gives the hosts their physical IPs, sitting on eth0

I also have a physical network (which I named incusphy0) and is on eth1 for each node.

The idea I have is using Incus Projects with OVN networks that connect to incusphy0, providing separated tenants.

Naturally, I’d want to have the ovn-central CTs on incusbr0, as part of the management plane but it looks like the easiest way to do that would be to port forward the OVN stuff between the hosts, and potentially add routes to my router. Switching incusbr0 to a physical network sounds like a great way to lock myself out of the nodes since there isn’t a local terminal I could use to recover.

The other option would be to put them on incusphy0, giving them direct access to the network.

Also, a MACVLAN might work too but thats just another thing to look into.

So I’m curious how you set up those ovn-central CTs (and other control plane CTs)

@NocturnalDanger Take a look at meshbr0. When I deployed an IncusOS cluster (using Operations Center), this network was created automatically together with the internal project intended for management containers for services such as OVN, Linstor and Ceph.

Normal bridge incusbr0 with NAT does not provide a direct connectivity for containers located on different cluster members, so ovn-central services would not be able to talk to each other directly. Using host IPs and port forwarding is possible (I did that with lots of help from AI assistents), but not easy as ovn-central, by default, tries to create network listeners for RAFT ports on the addresses of OVN cluster members, and fails if these are host IP addresses.

Note that meshbr0 uses only IPv6 SLAAC addresses, so OVN (and other services) need to be configured using IPv6 addresses.

Alternatively, management containers such as ovn-central can be connected directly to the host network.

Thanks for the info.

I did attach my ovn-central CTs to the physical uplink. I was hoping to keep them on a separate NIC so I could keep my management traffic and CT traffic isolated (Obviously OVN would need some firewall ports opened in the firewall for that). I looked at trying to route traffic through the NAT but decided it wasn’t worth the effort.

Ill look a bit more into that Wiki page you linked, but it looks like it might be a way for me to get them on the right NIC without the risk of losing access through my management IPs.