Recover of ovn network

Hello everyone,

I have lost one of my nodes in the cluster (tnode4) and setup all of the settings. But when I launch a container, get this error message.
Error: Failed instance creation: Failed pre-start check for device “eth0”: Network “ovnnet” unavailable on this server
I have backup of /etc and /var/lib/incus directories, how should I recover the ovnnet from those backups? can someone give any suggestions?
Regards.
The network settings of tnode1 and tnode4 are the same. Here are the settings as follows.


indiana@tnode1:~$ incus network ls
┌────────────────┬──────────┬─────────┬───────────────┬───────────────────────────┬─────────────┬─────────┬─────────┐
│      NAME      │   TYPE   │ MANAGED │     IPV4      │           IPV6            │ DESCRIPTION │ USED BY │  STATE  │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ UPLINK         │ physical │ YES     │               │                           │             │ 1       │ CREATED │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ br-ext         │ bridge   │ NO      │               │                           │             │ 1       │         │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ br-int         │ bridge   │ NO      │               │                           │             │ 0       │         │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ eth0           │ physical │ NO      │               │                           │             │ 0       │         │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ genev_sys_6081 │ unknown  │ NO      │               │                           │             │ 0       │         │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ incusovn5      │ bridge   │ NO      │               │                           │             │ 0       │         │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ incusovn5a     │ unknown  │ NO      │               │                           │             │ 0       │         │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ incusovn5b     │ unknown  │ NO      │               │                           │             │ 0       │         │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ lo             │ loopback │ NO      │               │                           │             │ 0       │         │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ ovnnet         │ ovn      │ YES     │ 10.1.224.1/24 │ fd42:8fa1:a0b5:8fb4::1/64 │             │ 9       │ CREATED │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ ovs-system     │ unknown  │ NO      │               │                           │             │ 0       │         │
└────────────────┴──────────┴─────────┴───────────────┴───────────────────────────┴─────────────┴─────────┴─────────┘
root@tnode4:~# incus network ls
┌────────────────┬──────────┬─────────┬───────────────┬───────────────────────────┬─────────────┬─────────┬─────────┐
│      NAME      │   TYPE   │ MANAGED │     IPV4      │           IPV6            │ DESCRIPTION │ USED BY │  STATE  │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ UPLINK         │ physical │ YES     │               │                           │             │ 1       │ CREATED │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ br-ext         │ bridge   │ NO      │               │                           │             │ 0       │         │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ br-int         │ bridge   │ NO      │               │                           │             │ 0       │         │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ eth0           │ physical │ NO      │               │                           │             │ 0       │         │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ genev_sys_6081 │ unknown  │ NO      │               │                           │             │ 0       │         │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ incusovn5      │ bridge   │ NO      │               │                           │             │ 0       │         │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ lo             │ loopback │ NO      │               │                           │             │ 0       │         │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ ovnnet         │ ovn      │ YES     │ 10.1.224.1/24 │ fd42:8fa1:a0b5:8fb4::1/64 │             │ 10      │ CREATED │
├────────────────┼──────────┼─────────┼───────────────┼───────────────────────────┼─────────────┼─────────┼─────────┤
│ ovs-system     │ unknown  │ NO      │               │                           │             │ 0       │         │
└────────────────┴──────────┴─────────┴───────────────┴───────────────────────────┴─────────────┴─────────┴─────────┘

I hope someone else can help you, if not I offer free remote hands.

A few things to look at:

  • incus cluster list so we can see what we’re working with exactly
  • incus config show --target NAME for each of the servers
  • ovs-vsctl show on each of the servers
  • incus network show ovnnet on any of the servers
  • incus network show UPLINK --target NAME for each of the servers

Thanks @stgraber and @jochumdev for the valuable reply, I have mixed up my tnode host a little bit and restore the whole /var/lib/incus directory from the borg2 backup. And when I start up the incus service, I get this error message.

indiana@tnode4:~$ systemctl status incus
● incus.service - Incus - Daemon
     Loaded: loaded (/usr/lib/systemd/system/incus.service; indirect; preset: enabled)
     Active: deactivating (stop-sigterm) (Result: timeout) since Mon 2026-08-31 21:58:56 +03; 32min ago
        Job: 1459
 Invocation: 574a85038b9546859cca640523caadbd
TriggeredBy: ● incus.socket
    Process: 3998 ExecStartPost=/opt/incus/lib/systemd/incusd waitready --timeout=600 (code=killed, signal=TERM)
   Main PID: 3997 (incusd)
      Tasks: 19
     Memory: 77.9M (peak: 111.8M)
        CPU: 1min 28.387s
     CGroup: /system.slice/incus.service
             └─3997 incusd --group incus-admin --logfile /var/log/incus/incusd.log

Aug 31 22:23:20 tnode4 incusd[3997]: time="2026-08-31T22:23:20+03:00" level=error msg="Failed connecting to global database" attempt=20 err="failed to create cowsql connection: no available cowsql leader server found"
Aug 31 22:24:21 tnode4 incusd[3997]: time="2026-08-31T22:24:21+03:00" level=error msg="Failed connecting to global database" attempt=25 err="failed to create cowsql connection: no available cowsql leader server found"
Aug 31 22:25:22 tnode4 incusd[3997]: time="2026-08-31T22:25:22+03:00" level=error msg="Failed connecting to global database" attempt=30 err="failed to create cowsql connection: no available cowsql leader server found"
Aug 31 22:26:24 tnode4 incusd[3997]: time="2026-08-31T22:26:24+03:00" level=error msg="Failed connecting to global database" attempt=35 err="failed to create cowsql connection: no available cowsql leader server found"
Aug 31 22:27:25 tnode4 incusd[3997]: time="2026-08-31T22:27:25+03:00" level=error msg="Failed connecting to global database" attempt=40 err="failed to create cowsql connection: no available cowsql leader server found"
Aug 31 22:28:25 tnode4 incusd[3997]: time="2026-08-31T22:28:25+03:00" level=error msg="Failed connecting to global database" attempt=45 err="failed to create cowsql connection: no available cowsql leader server found"
Aug 31 22:29:02 tnode4 systemd[1]: incus.service: start-post operation timed out. Terminating.
Aug 31 22:29:02 tnode4 systemd[1]: incus.service: Control process exited, code=killed, status=15/TERM
Aug 31 22:29:27 tnode4 incusd[3997]: time="2026-08-31T22:29:27+03:00" level=error msg="Failed connecting to global database" attempt=50 err="failed to create cowsql connection: no available cowsql leader server found"
Aug 31 22:30:28 tnode4 incusd[3997]: time="2026-08-31T22:30:28+03:00" level=error msg="Failed connecting to global database" attempt=55 err="failed to create cowsql connection: no available cowsql leader server found"

The good news is, cluster seems okey when I looked at the status from another node.
Is there any way to fix this thing up?
Regards.

Try moving /var/lib/incus/database/global aside and replace it with a copy from one of the other servers.

Sorry for the late response,

I have fixed the ovn network on tnode4 and seems working now. Thanks for the reply.

Regards.

Humm, I think I acted a bit too soon. I have fixed the network problem then restore the /var/lib/incus directory but still get the following message.

time="2026-09-03T00:33:34+03:00" level=error msg="Failed connecting to global database" attempt=6 err="failed to create cowsql connection: no available cowsql leader server found"

P.S. I have copied /var/lib/incus/database/global directory from another cluster member but still the same.

Hmm and the rest of the cluster is happy?
If so, you may want to grab sqlite3 /var/lib/incus/database/local.db "SELECT * FROM raft_nodes" from that server and from one of the working ones to compare.

Here are the latest states and outputs:

indiana@tnode3:~$ incus cluster ls
┌────────┬────────────────────────────┬──────────────────┬──────────────┬────────────────┬─────────────┬────────┬───────────────────┐
│  NAME  │            URL             │      ROLES       │ ARCHITECTURE │ FAILURE DOMAIN │ DESCRIPTION │ STATUS │      MESSAGE      │
├────────┼────────────────────────────┼──────────────────┼──────────────┼────────────────┼─────────────┼────────┼───────────────────┤
│ tnode1 │ https://192.168.1.201:8443 │ database         │ aarch64      │ default        │             │ ONLINE │ Fully operational │
├────────┼────────────────────────────┼──────────────────┼──────────────┼────────────────┼─────────────┼────────┼───────────────────┤
│ tnode2 │ https://192.168.1.202:8443 │ database-leader  │ aarch64      │ default        │             │ ONLINE │ Fully operational │
│        │                            │ database         │              │                │             │        │                   │
├────────┼────────────────────────────┼──────────────────┼──────────────┼────────────────┼─────────────┼────────┼───────────────────┤
│ tnode3 │ https://192.168.1.203:8443 │ database         │ aarch64      │ default        │             │ ONLINE │ Fully operational │
├────────┼────────────────────────────┼──────────────────┼──────────────┼────────────────┼─────────────┼────────┼───────────────────┤
│ tnode4 │ https://192.168.1.204:8443 │ database-standby │ aarch64      │ default        │             │ ONLINE │ Fully operational │
└────────┴────────────────────────────┴──────────────────┴──────────────┴────────────────┴─────────────┴────────┴───────────────────┘

indiana@tnode1$ sudo sqlite3 /var/lib/incus/database/local.db "SELECT * FROM raft_nodes"
[sudo] password for indiana: 
1|192.168.1.201:8443|0|tnode1
2|192.168.1.202:8443|0|tnode2
3|192.168.1.203:8443|0|tnode3
6|192.168.1.204:8443|1|tnode4

indiana@tnode4:~$ sudo sqlite3 /var/lib/incus/database/local.db "SELECT * FROM raft_nodes"
[sudo] password for indiana:
1|192.168.1.201:8443|0|tnode1
2|192.168.1.202:8443|0|tnode2
3|192.168.1.203:8443|0|tnode3
6|192.168.1.204:8443|1|tnode4

Hmm, so the cluster sees and is happy with tnode4 but tnode4 itself isn’t?
What happens if you do incus info --target tnode4 from tnode1?

And then from what you’ve said, I’m assuming that incus cluster list and incus info --target tnode1 from tnode4 both fail?

indiana@tnode1:~$ incus info --target tnode4
Error: Target cluster member is unavailable

indiana@tnode4:~$ sudo systemctl status incus
[sudo] password for indiana:
● incus.service - Incus - Daemon
Loaded: loaded (/usr/lib/systemd/system/incus.service; indirect; preset: enabled)
Active: activating (start-post) since Thu 2026-09-03 16:36:17 +03; 9min ago
Job: 6645
Invocation: fd596f1f3edd493ba81aaaf57cc9bdb9
TriggeredBy: ● incus.socket
Main PID: 7229 (incusd); Control PID: 7230 (incusd)
Tasks: 29
Memory: 81.4M (peak: 111.3M)
CPU: 1min 6.496s
CGroup: /system.slice/incus.service
├─7229 incusd --group incus-admin --logfile /var/log/incus/incusd.log
└─.control
└─7230 incusd waitready --timeout=600

Sep 03 16:38:43 tnode4 incusd[7229]: time="2026-09-03T16:38:43+03:00" level=error msg="Failed connecting to global database" attempt=11 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:38:55 tnode4 incusd[7229]: time="2026-09-03T16:38:55+03:00" level=error msg="Failed connecting to global database" attempt=12 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:39:07 tnode4 incusd[7229]: time="2026-09-03T16:39:07+03:00" level=error msg="Failed connecting to global database" attempt=13 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:39:19 tnode4 incusd[7229]: time="2026-09-03T16:39:19+03:00" level=error msg="Failed connecting to global database" attempt=14 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:39:31 tnode4 incusd[7229]: time="2026-09-03T16:39:31+03:00" level=error msg="Failed connecting to global database" attempt=15 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:40:32 tnode4 incusd[7229]: time="2026-09-03T16:40:32+03:00" level=error msg="Failed connecting to global database" attempt=20 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:41:34 tnode4 incusd[7229]: time="2026-09-03T16:41:34+03:00" level=error msg="Failed connecting to global database" attempt=25 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:42:36 tnode4 incusd[7229]: time="2026-09-03T16:42:36+03:00" level=error msg="Failed connecting to global database" attempt=30 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:43:37 tnode4 incusd[7229]: time="2026-09-03T16:43:37+03:00" level=error msg="Failed connecting to global database" attempt=35 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:44:38 tnode4 incusd[7229]: time="2026-09-03T16:44:38+03:00" level=error msg="Failed connecting to global database" attempt=40 err="failed to create cowsql connection: no available cowsql leader server found"

The commands that you shared stucks cos service is not running right now.

Hmm, but then why is the cluster reporting it as operational… Something must be answering the heartbeats…

On tnode1, can you show incus config show and incus admin sql global "SELECT * FROM nodes"?

indiana@tnode1:~$ incus config show
config:
backups.compression_algorithm: zstd
cluster.https_address: 192.168.1.201:8443
cluster.offline_threshold: "60"
core.https_address: 192.168.1.201:8443
core.metrics_address: :8444
logging.loki01.lifecycle.types: instance
logging.loki01.target.address: http://10.1.224.4:3100
logging.loki01.target.type: loki
network.ovn.northbound_connection: tcp:192.168.1.201:6641,tcp:192.168.1.202:6641,tcp:192.168.1.203:6641,tcp:192.168.1.204:6641
storage.backups_volume: local/backuppool
storage.images_volume: local/imagepool

indiana@tnode1:~$ incus admin sql global "SELECT * FROM nodes"
┌────┬────────┬─────────────┬────────────────────┬────────┬────────────────┬─────────────────────────────────────┬───────┬──────┬───────────────────┐
│ id │  name  │ description │      address       │ schema │ api_extensions │              heartbeat              │ state │ arch │ failure_domain_id │
├────┼────────┼─────────────┼────────────────────┼────────┼────────────────┼─────────────────────────────────────┼───────┼──────┼───────────────────┤
│ 1  │ tnode1 │             │ 192.168.1.201:8443 │ 77     │ 553            │ 2026-09-03T16:48:49.761292742+03:00 │ 0     │ 4    │              │
├────┼────────┼─────────────┼────────────────────┼────────┼────────────────┼─────────────────────────────────────┼───────┼──────┼───────────────────┤
│ 2  │ tnode2 │             │ 192.168.1.202:8443 │ 77     │ 553            │ 2026-09-03T16:48:40.008967591+03:00 │ 0     │ 4    │              │
├────┼────────┼─────────────┼────────────────────┼────────┼────────────────┼─────────────────────────────────────┼───────┼──────┼───────────────────┤
│ 3  │ tnode3 │             │ 192.168.1.203:8443 │ 77     │ 553            │ 2026-09-03T16:49:05.929943412+03:00 │ 0     │ 4    │              │
├────┼────────┼─────────────┼────────────────────┼────────┼────────────────┼─────────────────────────────────────┼───────┼──────┼───────────────────┤
│ 6  │ tnode4 │             │ 192.168.1.204:8443 │ 77     │ 553            │ 2026-09-03T16:48:48.146362926+03:00 │ 0     │ 4    │              │
└────┴────────┴─────────────┴────────────────────┴────────┴────────────────┴─────────────────────────────────────┴───────┴──────┴───────────────────┘

Okay, so something is definitely responding to heartbeats…
Can you try nc 192.168.1.204 8443 to see if something’s responding?

Also, may be worth checking that all your servers have their clocks in sync.

I tried from tnode1, tnode2 and tnode3.

nc 192.168.1.204 8443 command stucks, no response.

I have checked all of the time on the cluster member is okey.

Stuck isn’t good, it should instantly fail with something like:

stgraber@vorash-d13:~$ nc -v 192.168.1.204 8443
nc: connect to 192.168.1.204 port 8443 (tcp) failed: Connection refused

It getting stuck if not run with -v may actually suggest that something is responding which would be pretty inconsistent with your server not having a running incusd on it.

Interesting, the service looks running but service has problem.

indiana@tnode4:~$ ps -ef | grep -i incusd
root        8071       1 11 19:11 ?        00:01:00 incusd --group incus-admin --logfile /var/log/incus/incusd.log
root        8072       1  0 19:11 ?        00:00:00 incusd waitready --timeout=600
indiana     8269    1748  0 19:20 pts/0    00:00:00 grep -i incusd
indiana@tnode4:~$
indiana@tnode4:~$ systemctl status incus
● incus.service - Incus - Daemon
Loaded: loaded (/usr/lib/systemd/system/incus.service; indirect; preset: enabled)
Active: activating (start-post) since Thu 2026-09-03 19:11:23 +03; 9min ago
Job: 7534
Invocation: 5d774f7e1d3745c0b73a8cafcdbd9a89
TriggeredBy: ● incus.socket
Main PID: 8071 (incusd); Control PID: 8072 (incusd)
Tasks: 29
Memory: 81.8M (peak: 111.1M)
CPU: 1min 2.510s
CGroup: /system.slice/incus.service
├─8071 incusd --group incus-admin --logfile /var/log/incus/incusd.log
└─.control
└─8072 incusd waitready --timeout=600

Sep 03 19:13:48 tnode4 incusd[8071]: time="2026-09-03T19:13:48+03:00" level=error msg="Failed connecting to global database" attempt=11 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 19:14:01 tnode4 incusd[8071]: time="2026-09-03T19:14:01+03:00" level=error msg="Failed connecting to global database" attempt=12 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 19:14:13 tnode4 incusd[8071]: time="2026-09-03T19:14:13+03:00" level=error msg="Failed connecting to global database" attempt=13 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 19:14:25 tnode4 incusd[8071]: time="2026-09-03T19:14:25+03:00" level=error msg="Failed connecting to global database" attempt=14 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 19:14:38 tnode4 incusd[8071]: time="2026-09-03T19:14:38+03:00" level=error msg="Failed connecting to global database" attempt=15 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 19:15:41 tnode4 incusd[8071]: time="2026-09-03T19:15:41+03:00" level=error msg="Failed connecting to global database" attempt=20 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 19:16:43 tnode4 incusd[8071]: time="2026-09-03T19:16:43+03:00" level=error msg="Failed connecting to global database" attempt=25 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 19:17:46 tnode4 incusd[8071]: time="2026-09-03T19:17:46+03:00" level=error msg="Failed connecting to global database" attempt=30 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 19:18:47 tnode4 incusd[8071]: time="2026-09-03T19:18:47+03:00" level=error msg="Failed connecting to global database" attempt=35 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 19:19:49 tnode4 incusd[8071]: time="2026-09-03T19:19:49+03:00" level=error msg="Failed connecting to global database" attempt=40 err="failed to create cowsql connection: no available cowsql leader server found"


indiana@tnode1:~$ nc -v 192.168.1.204 8443
Ncat: Version 7.95 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.1.204:8443.

indiana@tnode1:~$ nc -zv 192.168.1.204 8443
Ncat: Version 7.95 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.1.204:8443.
Ncat: 0 bytes sent, 0 bytes received in 0.16 seconds.

I think that I found the problem but not sure it is related with the cowsql. I get the following error message when I looked at the incus service on the other nodes that are tnode1,tnode2,tnode3.
level=error msg=“Invalid client certificate CN=root@tnode4,O=Linux Containers (cdb5da20608c06f6ecf0b86ecab2618d19b41a0b58579c04e41fa7f88350f3f3) from 192.168.1.204:37886”
Is there any easy way to fix it or do i need to delete the tnode4 from the cluster and join it again?

Regards.

Did you do anything that would have changed /var/lib/incus/server.crt and /var/lib/incus/server.key on tnode4?

Can you show incus config trust list to see what’s in there currently?

As far as I remember, I had problem with lvm storage with metadata and delete the /var/lib/incus directory and tried to reinstall it but never succeeded at the end I restored from the old backup.

indiana@tnode1:~$ incus config trust list
┌────────────────┬────────┬─────────────┬──────────────┬──────────────────────┐
│ NAME │ TYPE │ DESCRIPTION │ FINGERPRINT │ EXPIRY DATE │
├────────────────┼────────┼─────────────┼──────────────┼──────────────────────┤
│ indiana@debian │ client │ │ 22a880ef01b1 │ 2036/01/11 20:45 +03 │
├────────────────┼────────┼─────────────┼──────────────┼──────────────────────┤
│ tnode1 │ server │ │ 08488766685b │ 2036/05/18 08:35 +03 │
├────────────────┼────────┼─────────────┼──────────────┼──────────────────────┤
│ tnode2 │ server │ │ 7d7e2c3c01fc │ 2036/05/18 09:52 +03 │
├────────────────┼────────┼─────────────┼──────────────┼──────────────────────┤
│ tnode3 │ server │ │ 98787c2fb56f │ 2036/05/18 11:23 +03 │
├────────────────┼────────┼─────────────┼──────────────┼──────────────────────┤
│ tnode4 │ server │ │ 74ff95ed3a3b │ 2036/08/27 16:32 +03 │
└────────────────┴────────┴─────────────┴──────────────┴──────────────────────┘