Standalone OVN network on IncusOS

I am trying to setup OVN on a single (non-clustered) host running IncusOS.

The main purpose is to use isolated project-specific networks (features.networks=true) for wich OVN networks seems to be the only option.

I adapted the Incus OVN tutorial to IncusOS using this comment from @stgraber :

Now for OVN on IncusOS, the way this works is that IncusOS runs both OVS and the OVN controller (ovn-host). The control plane (ovn-central) should be run in containers or VMs, possibly on the same cluster, just so long as they’re not themselves running on an OVN network.

Specifically:

  1. Created an onv-central container:
incus launch images:ubuntu/24.04 ovn-central
  1. Installed the ovn-central package:
incus exec ovn-central -- apt update
incus exec ovn-central -- apt install ovn-central
  1. Ensured that southbound and northbound databases listen to connections an all interfaces:
incus exec ovn-central -- vi /etc/default/ovn-central

and pasted:

OVN_CTL_OPTS="--db-nb-addr=0.0.0.0 --db-nb-create-insecure-remote=yes --db-sb-addr=0.0.0.0 --db-sb-create-insecure-remote=yes"
  1. Configured the OVN service of IncusOS where <ovn-ceintral-IP> is the IP address of the ovn-central container:
incus admin os service edit ovn 

and pasted:

config:
  database: tcp:<ovn-ceintral-IP>:6642
  enabled: true
  tunnel_address: ""
  tunnel_protocol: geneve
state: {}
  1. Configured the communication between Incus and OVN northbound database:
incus config set network.ovn.northbound_connection=tcp:<ovn-ceintral-IP>:6641

The rest is standard:

incus network create ovntest  --type=ovn network=UPLINK
incus launch images:debian/13 c1 --network ovntest  

The OVN network ovntest is created fine and the container c1 gets network connectivity.

The problem: when the IncusOS host reboots, the shutdown process hangs for 15 minutes. The output of incus admin os debug log does not show many details during this time:

[2026/08/25 15:39:57 WEST] systemd-networkd: incusbr0: Link DOWN
[2026/08/25 15:39:57 WEST] dnsmasq: error binding DHCP socket to device incusbr0
[2026/08/25 15:39:57 WEST] dnsmasq-dhcp: router advertisement on fd42:f806:c351:2734::,
[2026/08/25 15:39:57 WEST] systemd: var-lib-incus-storage\x2dpools-local-containers-c1.mount: Deactivated successfully.
[2026/08/25 15:49:26 WEST] systemd: incus-startup.service: Stopping timed out. Terminating.
[2026/08/25 15:49:26 WEST] systemd: incus-startup.service: Control process exited, code=killed, status=15/TERM
[2026/08/25 15:49:26 WEST] systemd: incus-startup.service: Failed with result 'timeout'.
[2026/08/25 15:49:26 WEST] systemd: Stopped incus-startup.service - Incus - Startup check.
[2026/08/25 15:49:26 WEST] kernel: kauditd_printk_skb: 62 callbacks suppressed
[2026/08/25 15:49:26 WEST] kernel: audit: type=1131 audit(1787669366.635:720): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=incus-startup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[2026/08/25 15:49:26 WEST] incusd: time="2026-08-25T14:49:26Z" level=warning msg="Ignoring signal, shutdown already in progress" signal=terminated
[2026/08/25 15:49:26 WEST] systemd: Stopping incus.service - Incus - Daemon...
[2026/08/25 15:54:56 WEST] systemd: incus.service: State 'stop-sigterm' timed out. Killing.
[2026/08/25 15:54:56 WEST] systemd: incus.service: Killing process 1454 (incusd) with signal SIGKILL.
[2026/08/25 15:54:56 WEST] systemd: incus.service: Main process exited, code=killed, status=9/KILL
[2026/08/25 15:54:56 WEST] systemd: incus.service: Failed with result 'timeout'.
[2026/08/25 15:54:56 WEST] systemd: incus.service: Unit process 1611 (dnsmasq) remains running after unit stopped.
[2026/08/25 15:54:56 WEST] systemd: Stopped incus.service - Incus - Daemon.
[2026/08/25 15:54:56 WEST] systemd: incus.service: Consumed 9min 410ms CPU time, 1.3G memory peak.

The symptoms are very similar to those described in this post.

The system hangs even if I delete container c1 (but not the ovntest network) and stop the container ovn-central manually before reboot.

My impression is that the ovn services run by IncusOS (ovn-controller) require ovn databases run by ovn-central to be reachable during the shutdown process. This works for the clustered OVN setup where ovn-controller of other cluster members should answer the requests, but for a standalone OVN setup, the OVN databases becomes unreachable once the ovn-central container stops.

Is there any solution to this problem?

Hmm I have ovn in my nested setup script , I don’t have any issues with it, maybe that helps you to debug?

Good to know is: that script disables AppArmor in the nested because I had troubles with it from my host.

The issue is not with Incus but IncusOS.

Just making sure, you’re rebooting using incus admin os system reboot?

Yes, this is the only way to reboot as I do not have a physical access to the host. I also tried rebooting from the Incus-ui, but this method uses the same incus os API as far as I understand.

I also tried incus admin os application restart incus and this did not get stuck. I am not sure however if that also restarts the ovn service.

On my other IncusOS cluster the ovn service configured similarly using containers does not seem to exhibit this problem, but I did not try to shutdown all cluster hosts to make ovn-central databases unreachable.

If that is not an expected behavior, what else could be done to diagnose the problem?

I’m taking a look.

Many thanks for a quick patch @stgraber! The reboot time has now reduced considerably!

Do I understand it correctly, that the OVN networks are still forcefully brought down, just with a smaller timeout (of about 1-2 minutes) per network? So, in case there are many OVN networks, these timeouts can accumulate as shown in the log below:

[2026/09/05 11:27:04 CEST] ovn-controller: ovs|00105|reconnect|INFO|ssl:10.99.0.3:6642: connection attempt failed (No route to host)
[2026/09/05 11:27:04 CEST] ovn-controller: ovs|00106|reconnect|INFO|ssl:10.99.0.3:6642: continuing to reconnect in the background but suppressing further logging
[2026/09/05 11:27:10 CEST] ovs-vswitchd: ovs|00086|connmgr|INFO|br-int<->unix#3: 1597 flow_mods in the 38 s starting 56 s ago (1435 adds, 162 deletes)
[2026/09/05 11:27:23 CEST] ovn-controller: 2026-09-05T09:27:23Z|00107|memory_trim|INFO|Detected inactivity (last active 30003 ms ago): trimming memory
[2026/09/05 11:27:23 CEST] ovn-controller: ovs|00107|memory_trim|INFO|Detected inactivity (last active 30003 ms ago): trimming memory
[2026/09/05 11:28:36 CEST] incusd: time="2026-09-05T09:28:36Z" level=error msg="Failed to bring down network" err="Failed deleting OVS chassis \"c56cfa26-ea3f-4b3d-895d-e8ef28249e92\" from chassis group \"incus-net39\": context deadline exceeded: while awaiting reconnection" name=myovn1 project=default
[2026/09/05 11:30:08 CEST] incusd: time="2026-09-05T09:30:08Z" level=error msg="Failed to bring down network" err="Failed deleting OVS chassis \"c56cfa26-ea3f-4b3d-895d-e8ef28249e92\" from chassis group \"incus-net40\": context deadline exceeded: while awaiting reconnection" name=myovn2 project=default
[2026/09/05 11:31:40 CEST] incusd: time="2026-09-05T09:31:40Z" level=error msg="Failed to bring down network" err="Failed deleting OVS chassis \"c56cfa26-ea3f-4b3d-895d-e8ef28249e92\" from chassis group \"incus-net41\": context deadline exceeded: while awaiting reconnection" name=myovn3 project=default
[2026/09/05 11:31:40 CEST] systemd-networkd: ovn: Link DOWN

Is it possible to tear down the networks more gracefully (without error messages) or reduce the overall timeout in some other way? E.g., set timeout to zero during the shut down process or shut down OVN networks asynchronously?

We may be able to tear them down in parallel to save time, but any system that’s busy with a lot of networks is also likely to be a cluster with an actual HA deployment of OVN which wouldn’t run into this issue to begin with :slight_smile:

The error is absolutely correct though. When we bring an Incus server down, we want to remove it from the list of validate candidates for OVN active chassis. We’re not going to fail on startup if the entry is already there, but us not being able to clear it on shutdown is definitely an issue and one we should log.