I have lost one of my nodes in the cluster (tnode4) and setup all of the settings. But when I launch a container, get this error message.
Error: Failed instance creation: Failed pre-start check for device “eth0”: Network “ovnnet” unavailable on this server
I have backup of /etc and /var/lib/incus directories, how should I recover the ovnnet from those backups? can someone give any suggestions?
Regards.
The network settings of tnode1 and tnode4 are the same. Here are the settings as follows.
Thanks @stgraber and @jochumdev for the valuable reply, I have mixed up my tnode host a little bit and restore the whole /var/lib/incus directory from the borg2 backup. And when I start up the incus service, I get this error message.
indiana@tnode4:~$ systemctl status incus
● incus.service - Incus - Daemon
Loaded: loaded (/usr/lib/systemd/system/incus.service; indirect; preset: enabled)
Active: deactivating (stop-sigterm) (Result: timeout) since Mon 2026-08-31 21:58:56 +03; 32min ago
Job: 1459
Invocation: 574a85038b9546859cca640523caadbd
TriggeredBy: ● incus.socket
Process: 3998 ExecStartPost=/opt/incus/lib/systemd/incusd waitready --timeout=600 (code=killed, signal=TERM)
Main PID: 3997 (incusd)
Tasks: 19
Memory: 77.9M (peak: 111.8M)
CPU: 1min 28.387s
CGroup: /system.slice/incus.service
└─3997 incusd --group incus-admin --logfile /var/log/incus/incusd.log
Aug 31 22:23:20 tnode4 incusd[3997]: time="2026-08-31T22:23:20+03:00" level=error msg="Failed connecting to global database" attempt=20 err="failed to create cowsql connection: no available cowsql leader server found"
Aug 31 22:24:21 tnode4 incusd[3997]: time="2026-08-31T22:24:21+03:00" level=error msg="Failed connecting to global database" attempt=25 err="failed to create cowsql connection: no available cowsql leader server found"
Aug 31 22:25:22 tnode4 incusd[3997]: time="2026-08-31T22:25:22+03:00" level=error msg="Failed connecting to global database" attempt=30 err="failed to create cowsql connection: no available cowsql leader server found"
Aug 31 22:26:24 tnode4 incusd[3997]: time="2026-08-31T22:26:24+03:00" level=error msg="Failed connecting to global database" attempt=35 err="failed to create cowsql connection: no available cowsql leader server found"
Aug 31 22:27:25 tnode4 incusd[3997]: time="2026-08-31T22:27:25+03:00" level=error msg="Failed connecting to global database" attempt=40 err="failed to create cowsql connection: no available cowsql leader server found"
Aug 31 22:28:25 tnode4 incusd[3997]: time="2026-08-31T22:28:25+03:00" level=error msg="Failed connecting to global database" attempt=45 err="failed to create cowsql connection: no available cowsql leader server found"
Aug 31 22:29:02 tnode4 systemd[1]: incus.service: start-post operation timed out. Terminating.
Aug 31 22:29:02 tnode4 systemd[1]: incus.service: Control process exited, code=killed, status=15/TERM
Aug 31 22:29:27 tnode4 incusd[3997]: time="2026-08-31T22:29:27+03:00" level=error msg="Failed connecting to global database" attempt=50 err="failed to create cowsql connection: no available cowsql leader server found"
Aug 31 22:30:28 tnode4 incusd[3997]: time="2026-08-31T22:30:28+03:00" level=error msg="Failed connecting to global database" attempt=55 err="failed to create cowsql connection: no available cowsql leader server found"
The good news is, cluster seems okey when I looked at the status from another node.
Is there any way to fix this thing up?
Regards.
Humm, I think I acted a bit too soon. I have fixed the network problem then restore the /var/lib/incus directory but still get the following message.
time="2026-09-03T00:33:34+03:00" level=error msg="Failed connecting to global database" attempt=6 err="failed to create cowsql connection: no available cowsql leader server found"
P.S. I have copied /var/lib/incus/database/global directory from another cluster member but still the same.
Hmm and the rest of the cluster is happy?
If so, you may want to grab sqlite3 /var/lib/incus/database/local.db "SELECT * FROM raft_nodes" from that server and from one of the working ones to compare.
indiana@tnode1:~$ incus info --target tnode4
Error: Target cluster member is unavailable
indiana@tnode4:~$ sudo systemctl status incus
[sudo] password for indiana:
● incus.service - Incus - Daemon
Loaded: loaded (/usr/lib/systemd/system/incus.service; indirect; preset: enabled)
Active: activating (start-post) since Thu 2026-09-03 16:36:17 +03; 9min ago
Job: 6645
Invocation: fd596f1f3edd493ba81aaaf57cc9bdb9
TriggeredBy: ● incus.socket
Main PID: 7229 (incusd); Control PID: 7230 (incusd)
Tasks: 29
Memory: 81.4M (peak: 111.3M)
CPU: 1min 6.496s
CGroup: /system.slice/incus.service
├─7229 incusd --group incus-admin --logfile /var/log/incus/incusd.log
└─.control
└─7230 incusd waitready --timeout=600
Sep 03 16:38:43 tnode4 incusd[7229]: time="2026-09-03T16:38:43+03:00" level=error msg="Failed connecting to global database" attempt=11 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:38:55 tnode4 incusd[7229]: time="2026-09-03T16:38:55+03:00" level=error msg="Failed connecting to global database" attempt=12 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:39:07 tnode4 incusd[7229]: time="2026-09-03T16:39:07+03:00" level=error msg="Failed connecting to global database" attempt=13 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:39:19 tnode4 incusd[7229]: time="2026-09-03T16:39:19+03:00" level=error msg="Failed connecting to global database" attempt=14 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:39:31 tnode4 incusd[7229]: time="2026-09-03T16:39:31+03:00" level=error msg="Failed connecting to global database" attempt=15 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:40:32 tnode4 incusd[7229]: time="2026-09-03T16:40:32+03:00" level=error msg="Failed connecting to global database" attempt=20 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:41:34 tnode4 incusd[7229]: time="2026-09-03T16:41:34+03:00" level=error msg="Failed connecting to global database" attempt=25 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:42:36 tnode4 incusd[7229]: time="2026-09-03T16:42:36+03:00" level=error msg="Failed connecting to global database" attempt=30 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:43:37 tnode4 incusd[7229]: time="2026-09-03T16:43:37+03:00" level=error msg="Failed connecting to global database" attempt=35 err="failed to create cowsql connection: no available cowsql leader server found"
Sep 03 16:44:38 tnode4 incusd[7229]: time="2026-09-03T16:44:38+03:00" level=error msg="Failed connecting to global database" attempt=40 err="failed to create cowsql connection: no available cowsql leader server found"
The commands that you shared stucks cos service is not running right now.
Stuck isn’t good, it should instantly fail with something like:
stgraber@vorash-d13:~$ nc -v 192.168.1.204 8443
nc: connect to 192.168.1.204 port 8443 (tcp) failed: Connection refused
It getting stuck if not run with -v may actually suggest that something is responding which would be pretty inconsistent with your server not having a running incusd on it.
I think that I found the problem but not sure it is related with the cowsql. I get the following error message when I looked at the incus service on the other nodes that are tnode1,tnode2,tnode3.
level=error msg=“Invalid client certificate CN=root@tnode4,O=Linux Containers (cdb5da20608c06f6ecf0b86ecab2618d19b41a0b58579c04e41fa7f88350f3f3) from 192.168.1.204:37886”
Is there any easy way to fix it or do i need to delete the tnode4 from the cluster and join it again?
As far as I remember, I had problem with lvm storage with metadata and delete the /var/lib/incus directory and tried to reinstall it but never succeeded at the end I restored from the old backup.