IncusOS OVN Issues

IncusOS
1

Hello!

I’m trying to enable the OVN service on IncusOS.

Just as background, I am running these commands in Powershell

The first thing I noticed that incus admin os service show ovn returns a YAML file and incus admin os service edit ovn takes a JSON file.

So I used incus query /os/1.0/services/ovn > ovn.json, which gave me a JSON and created a new file.

I edited it to be:

{
	"config": {
		"database": "tcp:10.40.0.2:6642",
		"enabled": true,
		"ic_chassis": false,
		"tls_ca_certificate": "",
		"tls_client_certificate": "",
		"tls_client_key": "",
		"tunnel_address": "10.40.0.2",
		"tunnel_protocol": "geneve"
	},
	"state": {}
}

Reviewing the IncusOS OVN Documentation and the Incus OVN Documentation, I came up with the assumptions:

  1. database = external_ids:ovn-remote
  2. tunnel_address = external_ids:ovn-encap-ip
  3. tunnel_protocol = external_ids:ovn-encap-type

I get the output (new lines added for readability):

PS C:\Users\Chase\Downloads> Get-Content .\ovn.json | incus admin os service edit ovn
WARNING: The IncusOS API and configuration is subject to change

Error: Failed to run: 
ovs-vsctl set open_vswitch . 
external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 
external_ids:ovn-remote=tcp:10.40.0.2:6642 
external_ids:ovn-encap-type=geneve 
external_ids:ovn-encap-ip=10.40.0.2 
external_ids:ovn-is-interconn=false: exit status 1 (ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory))
PS C:\Users\Chase\Downloads>

I also noticed in the Incus OS Documentation, it has a different example with external_ids:ovn-remote=unix:/run/ovn/ovnsb_db.sock and external_ids:ovn-encap-ip=127.0.0.1. Using that example, I’d get:

{
	"config": {
		"database": "unix:/run/openvswitch/db.sock",
		"enabled": true,
		"ic_chassis": false,
		"tls_ca_certificate": "",
		"tls_client_certificate": "",
		"tls_client_key": "",
		"tunnel_address": "127.0.0.1",
		"tunnel_protocol": "geneve"
	},
	"state": {}
}

and

PS C:\Users\Chase\Downloads> Get-Content .\ovn.json | incus admin os service edit ovn
WARNING: The IncusOS API and configuration is subject to change

Error: Failed to run: 
ovs-vsctl set open_vswitch . 
external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 
external_ids:ovn-remote=unix:/run/openvswitch/db.sock 
external_ids:ovn-encap-type=geneve 
external_ids:ovn-encap-ip=127.0.0.1 
external_ids:ovn-is-interconn=false: exit status 1 (ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory))
PS C:\Users\Chase\Downloads>

Is there something I’m doing wrong?

2

IncusOS and OVN is something that I’ve implemented but then haven’t yet rolled out in the validation lab, so there’s quite a good chance that this is broken right now.

I’ve put a TODO list item for it for hopefully later today so I can try to reproduce the issue and get this working reliably in the validation lab.

1 Like
3

So I nuked the endpoint and reinstalled the OS, from scratch, and instead of using Windows Powershell, I used WSL with Debian.

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus remote add incus-02 10.40.0.2
Certificate fingerprint: [...]
ok (y/n/[fingerprint])? y
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus remote switch incus-02
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus network list
+----------+--------+---------+-----------------+---------------------------+----------------------------+---------+---------+
|   NAME   |  TYPE  | MANAGED |      IPV4       |           IPV6            |        DESCRIPTION         | USED BY |  STATE  |
+----------+--------+---------+-----------------+---------------------------+----------------------------+---------+---------+
| incusbr0 | bridge | YES     | 10.207.223.1/24 | fd42:fce8:7b36:1d77::1/64 | Local network bridge (NAT) | 1       | CREATED |
+----------+--------+---------+-----------------+---------------------------+----------------------------+---------+---------+
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus network edit incusbr0
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus network list
+----------+--------+---------+---------------+------+----------------------------+---------+---------+
|   NAME   |  TYPE  | MANAGED |     IPV4      | IPV6 |        DESCRIPTION         | USED BY |  STATE  |
+----------+--------+---------+---------------+------+----------------------------+---------+---------+
| incusbr0 | bridge | YES     | 10.100.0.2/24 | none | Local network bridge (NAT) | 1       | CREATED |
+----------+--------+---------+---------------+------+----------------------------+---------+---------+
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus admin os service show ovn
WARNING: The IncusOS API and configuration is subject to change

config:
  database: ""
  enabled: false
  ic_chassis: false
  tls_ca_certificate: ""
  tls_client_certificate: ""
  tls_client_key: ""
  tunnel_address: ""
  tunnel_protocol: ""
state: {}
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus admin os service edit ovn
WARNING: The IncusOS API and configuration is subject to change

Config parsing error: Failed to run: ovs-vsctl set open_vswitch . external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 external_ids:ovn-remote=tcp:10.40.0.2:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=10.40.0.2 external_ids:ovn-is-interconn=false: exit status 1 (ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory))
Press enter to open the editor again or ctrl+c to abort change
^C
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus version
Client version: 6.19.1
Server version: 6.19.1
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config trust add-certificate ~/incus-ui.crt
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.ca_crt

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.client_cert

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.client_key

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.integration_bridge

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.northbound_connection

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.network.ovs.connection

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$

I think it’s interesting that those configs aren’t returning anything when they’re present in the WebUI

image
image1223×505 33.8 KB

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config set network.ovn.integration_bridge incusbr0
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config set network.ovn.northbound_connection unix:/run/ovn/ovnnb_db.sock
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config set network.ovn.network.ovs.connection unix:/run/openvswitch/db.sock
Error: cannot set 'network.ovn.network.ovs.connection' to 'unix:/run/openvswitch/db.sock': unknown key
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config set network.ovn.network.ovs.connection /run/openvswitch/db.sock
Error: cannot set 'network.ovn.network.ovs.connection' to '/run/openvswitch/db.sock': unknown key
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.integration_bridge
incusbr0
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.northbound_connection

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus config get network.ovn.network.ovs.connection

chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus admin os service edit ovn
WARNING: The IncusOS API and configuration is subject to change

Config parsing error: Failed to run: ovs-vsctl set open_vswitch . external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 external_ids:ovn-remote=tcp:10.40.0.2:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=10.40.0.2 external_ids:ovn-is-interconn=false: exit status 1 (ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory))
Press enter to open the editor again or ctrl+c to abort change

Config parsing error: Failed to run: ovs-vsctl set open_vswitch . external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 external_ids:ovn-remote=tcp:10.40.0.2:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=127.0.0.1 external_ids:ovn-is-interconn=false: exit status 1 (ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory))
Press enter to open the editor again or ctrl+c to abort change

Config parsing error: Failed to run: ovs-vsctl set open_vswitch . external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 external_ids:ovn-remote=tcp:10.40.0.2:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=127.0.0.1 external_ids:ovn-is-interconn=false: exit status 1 (ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory))
Press enter to open the editor again or ctrl+c to abort change
^C
chase@DESKTOP-2TUBIB8:/mnt/c/Users/Chase$ incus admin os service show ovn
WARNING: The IncusOS API and configuration is subject to change

config:
  database: tcp:10.40.0.2:6642
  enabled: true
  ic_chassis: false
  tls_ca_certificate: ""
  tls_client_certificate: ""
  tls_client_key: ""
  tunnel_address: 127.0.0.1
  tunnel_protocol: geneve
state: {}

When I poke the OVN service with the API:

chase@DESKTOP-2TUBIB8:~$ curl -k --cert curl-client.pem https://10.40.0.2:8443/os/1.0/services/ovn
{"type":"sync","status":"Success","status_code":200,"operation":"","error_code":0,"error":"","metadata":{"state":{},"config":{"enabled":true,"ic_chassis":false,"database":"tcp:10.40.0.2:6642","tls_client_certificate":"","tls_client_key":"","tls_ca_certificate":"","tunnel_address":"127.0.0.1","tunnel_protocol":"geneve"}}}
chase@DESKTOP-2TUBIB8:~$ curl -k --cert curl-client.pem https://10.40.0.2:8443/os/1.0/services/ovn/:reset
{"type":"error","status":"","status_code":0,"operation":"","error_code":501,"error":"not implemented","metadata":null}
chase@DESKTOP-2TUBIB8:~$

I get “not implemented”.

Heres the last couple entries for incus-osd.service:

chase@DESKTOP-2TUBIB8:~$ curl -k --cert curl-client.pem https://10.40.0.2:8443/os/1.0/debug/log | jq '[.metadata[] | select(._SYSTEMD_UNIT == "incus-osd.service")] | .[-5:]'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2093k    0 2093k    0     0  13.3M      0 --:--:-- --:--:-- --:--:-- 13.3M
[
  {
    "MESSAGE": "ovs|00002|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)",
    "PRIORITY": "3",
    "SYSLOG_FACILITY": "3",
    "SYSLOG_IDENTIFIER": "ovs-vsctl",
    "SYSLOG_TIMESTAMP": "Dec  3 23:39:16 ",
    "_BOOT_ID": "252c23c1ce714c118500cb920b5948b8",
    "_CAP_EFFECTIVE": "1ffffffffff",
    "_COMM": "ovs-vsctl",
    "_GID": "0",
    "_HOSTNAME": "f3bdd818-166e-11f0-996b-31eea3194a00",
    "_MACHINE_ID": "9ad4ea371459453aa524e32d60dc6567",
    "_PID": "2831",
    "_RUNTIME_SCOPE": "system",
    "_SELINUX_CONTEXT": "unconfined\n",
    "_SOURCE_REALTIME_TIMESTAMP": "1764805156303658",
    "_SYSTEMD_CGROUP": "/system.slice/incus-osd.service",
    "_SYSTEMD_INVOCATION_ID": "10d55b4fd1364c8d84a490c949f0b41a",
    "_SYSTEMD_SLICE": "system.slice",
    "_SYSTEMD_UNIT": "incus-osd.service",
    "_TRANSPORT": "syslog",
    "_UID": "0",
    "__CURSOR": "s=6ec19a00f3d046ffbba9c9f9d2afef56;i=94c;b=252c23c1ce714c118500cb920b5948b8;m=7213d766;t=64514b8708abd;x=e43c237a3be2eb28",
    "__MONOTONIC_TIMESTAMP": "1913902950",
    "__REALTIME_TIMESTAMP": "1764805156309693",
    "__SEQNUM": "2380",
    "__SEQNUM_ID": "6ec19a00f3d046ffbba9c9f9d2afef56"
  },
  {
    "MESSAGE": "ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 external_ids:ovn-remote=tcp:10.40.0.2:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=127.0.0.1 external_ids:ovn-is-interconn=false",
    "PRIORITY": "5",
    "SYSLOG_FACILITY": "3",
    "SYSLOG_IDENTIFIER": "ovs-vsctl",
    "SYSLOG_TIMESTAMP": "Dec  3 23:39:51 ",
    "_BOOT_ID": "252c23c1ce714c118500cb920b5948b8",
    "_CAP_EFFECTIVE": "1ffffffffff",
    "_COMM": "ovs-vsctl",
    "_EXE": "/usr/bin/ovs-vsctl",
    "_GID": "0",
    "_HOSTNAME": "f3bdd818-166e-11f0-996b-31eea3194a00",
    "_MACHINE_ID": "9ad4ea371459453aa524e32d60dc6567",
    "_PID": "2841",
    "_RUNTIME_SCOPE": "system",
    "_SELINUX_CONTEXT": "unconfined\n",
    "_SOURCE_REALTIME_TIMESTAMP": "1764805191885048",
    "_SYSTEMD_CGROUP": "/system.slice/incus-osd.service",
    "_SYSTEMD_INVOCATION_ID": "10d55b4fd1364c8d84a490c949f0b41a",
    "_SYSTEMD_SLICE": "system.slice",
    "_SYSTEMD_UNIT": "incus-osd.service",
    "_TRANSPORT": "syslog",
    "_UID": "0",
    "__CURSOR": "s=6ec19a00f3d046ffbba9c9f9d2afef56;i=94d;b=252c23c1ce714c118500cb920b5948b8;m=7432adcc;t=64514ba8f6122;x=f146e5892f167b78",
    "__MONOTONIC_TIMESTAMP": "1949478348",
    "__REALTIME_TIMESTAMP": "1764805191885090",
    "__SEQNUM": "2381",
    "__SEQNUM_ID": "6ec19a00f3d046ffbba9c9f9d2afef56"
  },
  {
    "MESSAGE": "ovs|00002|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)",
    "PRIORITY": "3",
    "SYSLOG_FACILITY": "3",
    "SYSLOG_IDENTIFIER": "ovs-vsctl",
    "SYSLOG_TIMESTAMP": "Dec  3 23:39:51 ",
    "_BOOT_ID": "252c23c1ce714c118500cb920b5948b8",
    "_CAP_EFFECTIVE": "1ffffffffff",
    "_COMM": "ovs-vsctl",
    "_EXE": "/usr/bin/ovs-vsctl",
    "_GID": "0",
    "_HOSTNAME": "f3bdd818-166e-11f0-996b-31eea3194a00",
    "_MACHINE_ID": "9ad4ea371459453aa524e32d60dc6567",
    "_PID": "2841",
    "_RUNTIME_SCOPE": "system",
    "_SELINUX_CONTEXT": "unconfined\n",
    "_SOURCE_REALTIME_TIMESTAMP": "1764805191885169",
    "_SYSTEMD_CGROUP": "/system.slice/incus-osd.service",
    "_SYSTEMD_INVOCATION_ID": "10d55b4fd1364c8d84a490c949f0b41a",
    "_SYSTEMD_SLICE": "system.slice",
    "_SYSTEMD_UNIT": "incus-osd.service",
    "_TRANSPORT": "syslog",
    "_UID": "0",
    "__CURSOR": "s=6ec19a00f3d046ffbba9c9f9d2afef56;i=94e;b=252c23c1ce714c118500cb920b5948b8;m=7432b017;t=64514ba8f636d;x=3f02b452385d9673",
    "__MONOTONIC_TIMESTAMP": "1949478935",
    "__REALTIME_TIMESTAMP": "1764805191885677",
    "__SEQNUM": "2382",
    "__SEQNUM_ID": "6ec19a00f3d046ffbba9c9f9d2afef56"
  },
  {
    "MESSAGE": "ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 external_ids:ovn-remote=tcp:10.40.0.2:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=127.0.0.1 external_ids:ovn-is-interconn=false",
    "PRIORITY": "5",
    "SYSLOG_FACILITY": "3",
    "SYSLOG_IDENTIFIER": "ovs-vsctl",
    "SYSLOG_TIMESTAMP": "Dec  3 23:40:02 ",
    "_BOOT_ID": "252c23c1ce714c118500cb920b5948b8",
    "_CAP_EFFECTIVE": "1ffffffffff",
    "_COMM": "ovs-vsctl",
    "_GID": "0",
    "_HOSTNAME": "f3bdd818-166e-11f0-996b-31eea3194a00",
    "_MACHINE_ID": "9ad4ea371459453aa524e32d60dc6567",
    "_PID": "2844",
    "_RUNTIME_SCOPE": "system",
    "_SELINUX_CONTEXT": "unconfined\n",
    "_SOURCE_REALTIME_TIMESTAMP": "1764805202789927",
    "_SYSTEMD_CGROUP": "/system.slice/incus-osd.service",
    "_SYSTEMD_INVOCATION_ID": "10d55b4fd1364c8d84a490c949f0b41a",
    "_SYSTEMD_SLICE": "system.slice",
    "_SYSTEMD_UNIT": "incus-osd.service",
    "_TRANSPORT": "syslog",
    "_UID": "0",
    "__CURSOR": "s=6ec19a00f3d046ffbba9c9f9d2afef56;i=94f;b=252c23c1ce714c118500cb920b5948b8;m=74d912ff;t=64514bb35c655;x=e1d97f5995b60dcc",
    "__MONOTONIC_TIMESTAMP": "1960383231",
    "__REALTIME_TIMESTAMP": "1764805202789973",
    "__SEQNUM": "2383",
    "__SEQNUM_ID": "6ec19a00f3d046ffbba9c9f9d2afef56"
  },
  {
    "MESSAGE": "ovs|00002|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)",
    "PRIORITY": "3",
    "SYSLOG_FACILITY": "3",
    "SYSLOG_IDENTIFIER": "ovs-vsctl",
    "SYSLOG_TIMESTAMP": "Dec  3 23:40:02 ",
    "_BOOT_ID": "252c23c1ce714c118500cb920b5948b8",
    "_CAP_EFFECTIVE": "1ffffffffff",
    "_COMM": "ovs-vsctl",
    "_GID": "0",
    "_HOSTNAME": "f3bdd818-166e-11f0-996b-31eea3194a00",
    "_MACHINE_ID": "9ad4ea371459453aa524e32d60dc6567",
    "_PID": "2844",
    "_RUNTIME_SCOPE": "system",
    "_SELINUX_CONTEXT": "unconfined\n",
    "_SOURCE_REALTIME_TIMESTAMP": "1764805202790040",
    "_SYSTEMD_CGROUP": "/system.slice/incus-osd.service",
    "_SYSTEMD_INVOCATION_ID": "10d55b4fd1364c8d84a490c949f0b41a",
    "_SYSTEMD_SLICE": "system.slice",
    "_SYSTEMD_UNIT": "incus-osd.service",
    "_TRANSPORT": "syslog",
    "_UID": "0",
    "__CURSOR": "s=6ec19a00f3d046ffbba9c9f9d2afef56;i=950;b=252c23c1ce714c118500cb920b5948b8;m=74d914bc;t=64514bb35c812;x=568dcb67bd4fc99d",
    "__MONOTONIC_TIMESTAMP": "1960383676",
    "__REALTIME_TIMESTAMP": "1764805202790418",
    "__SEQNUM": "2384",
    "__SEQNUM_ID": "6ec19a00f3d046ffbba9c9f9d2afef56"
  }
]
chase@DESKTOP-2TUBIB8:~$

I was hoping journalctl would have given me more information to work off of, hopefully it helps you though

Late night edit:
I also took a stab at just returning every unique “message”, where the JSON entry contained any of the four strings that I associate with OVN:

chase@DESKTOP-2TUBIB8:~$ curl -k --cert curl-client.pem https://10.40.0.2:8443/os/1.0/debug/log | jq '[.metadata[] | select(any(.[]; tostring | test("ovn|ovs|vswitch|vsctl"; "i")))]  | .[].MESSAGE' | sort | uniq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2093k    0 2093k    0     0  14.0M      0 --:--:-- --:--:-- --:--:-- 14.0M
"ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 external_ids:ovn-remote=tcp:10.40.0.2:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=10.40.0.2 external_ids:ovn-is-interconn=false"
"ovs|00001|vsctl|INFO|Called as ovs-vsctl set open_vswitch . external_ids:hostname=f3bdd818-166e-11f0-996b-31eea3194a00 external_ids:ovn-remote=tcp:10.40.0.2:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=127.0.0.1 external_ids:ovn-is-interconn=false"
"ovs|00002|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)"

Nothing too groundbreaking.

4

I was looking at the repo and in incus-osd/internal/services/service_ovn.go. I don’t know GO, I’m more familiar with other languages, but here’s what I’ve pieced together:

On line… (sigh)… 67, if the service isn’t enabled and the new config sets it to enable, it’ll run Start(ctx). Inside of Start(), at line 105, it enables the service for ovs-vswitch: systemd.StartUnit(ctx, "ovs-vswitchd.service").

The error I got is usually related to the ovsdb-server service not running.

If we reference line 90, which is run when OVN is disabled, it also stops a few other ovn-related services: systemd.StopUnit(ctx, "ovn-controller.service", "ovs-vswitchd.service", "ovsdb-server.service")

I was considering cloning the repo and testing my guess before commenting about it, but I haven’t had a chance yet today and I probably wont for another 8 hours.

My initial assumption is to change this:

	// Start OVS.
	err := systemd.StartUnit(ctx, "ovs-vswitchd.service")
	if err != nil {
		return err
	}

into this:

	// Start OVS.
	err := systemd.StartUnit(ctx, "ovn-controller.service", "ovs-vswitchd.service", "ovsdb-server.service")
	if err != nil {
		return err
	}

If I understand the codebase properly, thatll start the other OVN/OVS services that IncusOS is expecting to run. After OVS is started, line 101 enters the configure() function, and then line 135 (_, err := subprocess.RunCommand("ovs-vsctl", args...)) is what generates the error for the socket missing (presumably because ovsdb-server wasn’t running by this point.

5

Looking into this issue now. I’ve got it reproduced here.

6

Awesome, I havent had very much time to look at this. Ive looked through the source a few more times and im a bit more confident that my initial thought is probably part of the issue but i havent been able build the source and test it.

7
8

Latest testing build works here:

stgraber@castiana:~ (incus:dev-cluster/default)$ incus list
+------+---------+----------------------+-------------------------------------------------+-----------------+-----------+----------+
| NAME |  STATE  |         IPV4         |                      IPV6                       |      TYPE       | SNAPSHOTS | LOCATION |
+------+---------+----------------------+-------------------------------------------------+-----------------+-----------+----------+
| d13  | RUNNING | 10.20.162.2 (enp5s0) | fd42:e9ac:c3e5:65f:1266:6aff:fe98:50ac (enp5s0) | VIRTUAL-MACHINE | 0         | incus03  |
+------+---------+----------------------+-------------------------------------------------+-----------------+-----------+----------+
stgraber@castiana:~ (incus:dev-cluster/default)$ incus list --project internal
+--------------------+---------+------+------------------------------------------------+-----------+-----------+----------+
|        NAME        |  STATE  | IPV4 |                      IPV6                      |   TYPE    | SNAPSHOTS | LOCATION |
+--------------------+---------+------+------------------------------------------------+-----------+-----------+----------+
| linstor-controller | RUNNING |      | fd42:143d:672d:ff60:1266:6aff:fee5:7eb1 (eth0) | CONTAINER | 0         | incus01  |
+--------------------+---------+------+------------------------------------------------+-----------+-----------+----------+
| ovn-central        | RUNNING |      | fd42:143d:672d:ff60:1266:6aff:fe6c:e5ac (eth0) | CONTAINER | 0         | incus02  |
+--------------------+---------+------+------------------------------------------------+-----------+-----------+----------+
stgraber@castiana:~ (incus:dev-cluster/default)$ incus network list
+----------+--------+---------+-----------------+---------------------------+------------------------------+---------+---------+
|   NAME   |  TYPE  | MANAGED |      IPV4       |           IPV6            |         DESCRIPTION          | USED BY |  STATE  |
+----------+--------+---------+-----------------+---------------------------+------------------------------+---------+---------+
| default  | ovn    | YES     | 10.20.162.1/24  | fd42:e9ac:c3e5:65f::1/64  |                              | 1       | CREATED |
+----------+--------+---------+-----------------+---------------------------+------------------------------+---------+---------+
| incusbr0 | bridge | YES     | 10.167.140.1/24 | fd42:c9f0:6f69:913f::1/64 | Local network bridge (NAT)   | 2       | CREATED |
+----------+--------+---------+-----------------+---------------------------+------------------------------+---------+---------+
| meshbr0  | bridge | YES     | none            | fd42:143d:672d:ff60::/64  | Internal mesh network bridge | 3       | CREATED |
+----------+--------+---------+-----------------+---------------------------+------------------------------+---------+---------+
stgraber@castiana:~ (incus:dev-cluster/default)$ incus config show
config:
  cluster.https_address: '[2602:fc62:b:8007:1266:6aff:fe46:8bd]:8443'
  core.bgp_address: :149
  core.bgp_asn: "65001"
  core.bgp_routerid: 10.0.0.101
  core.https_address: :8443
  network.ovn.northbound_connection: tcp:[fd42:143d:672d:ff60:1266:6aff:fe6c:e5ac]:6641
  storage.backups_volume: local/backups
  storage.images_volume: local/images
  storage.linstor.controller_connection: http://[fd42:143d:672d:ff60:1266:6aff:fee5:7eb1]:3370
stgraber@castiana:~ (incus:dev-cluster/default)$ incus cluster list
+---------+-----------------------------------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
|  NAME   |                         URL                         |      ROLES      | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATUS |      MESSAGE      |
+---------+-----------------------------------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| incus01 | https://[2602:fc62:b:8007:1266:6aff:fe46:8bd]:8443  | database        | x86_64       | default        |             | ONLINE | Fully operational |
+---------+-----------------------------------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| incus02 | https://[2602:fc62:b:8007:1266:6aff:fe02:fa52]:8443 | database-leader | x86_64       | default        |             | ONLINE | Fully operational |
|         |                                                     | database        |              |                |             |        |                   |
+---------+-----------------------------------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| incus03 | https://[2602:fc62:b:8007:1266:6aff:fed2:70e9]:8443 | database        | x86_64       | default        |             | ONLINE | Fully operational |
+---------+-----------------------------------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
stgraber@castiana:~ (incus:dev-cluster/default)$

I need to do more manual tests of other aspects of that image before we push it to stable, but at least on the OVN front, it’s looking good.