incusOS fails to restart after update

IncusOS
1

I have a 3 node incusOS cluster. Looked at it this morning and all 3 nodes were down and only responding on the fall back port.

they all looked something like this:

image
image1920Ă—1581 411 KB

the log is mostly about the cluster breaking - stuff like this:

[2026/09/09 07:17:36 EDT] incusd: time="2026-09-09T11:17:36Z" level=warning msg="Cluster member isn't responding" name=i-host1
[2026/09/09 07:17:37 EDT] incusd: time="2026-09-09T11:17:37Z" level=warning msg="Cluster member isn't responding" name=i-host1
[2026/09/09 07:17:37 EDT] incusd: time="2026-09-09T11:17:37Z" level=warning msg="Cluster member isn't responding" name=i-host2
[2026/09/09 07:17:45 EDT] incusd: time="2026-09-09T11:17:45Z" level=warning msg="Cluster member isn't responding" name=i-host2
[2026/09/09 07:17:46 EDT] incusd: time="2026-09-09T11:17:46Z" level=warning msg="Cluster member isn't responding" name=i-host1
[2026/09/09 07:18:03 EDT] incusd: time="2026-09-09T11:18:03Z" level=warning msg="Failed to create warning" err="Failed to begin transaction: failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 07:18:45 EDT] incusd: time="2026-09-09T11:18:45Z" level=warning msg="Failed to create warning" err="Failed to begin transaction: context deadline exceeded"
[2026/09/09 07:19:17 EDT] incusd: time="2026-09-09T11:19:17Z" level=warning msg="Failed to get current cluster members" err="Failed to begin transaction: failed to create cowsql connection: no available cowsql leader server found" mode=normal
[2026/09/09 07:19:26 EDT] incusd: time="2026-09-09T11:19:26Z" level=error msg="Failed getting instance snapshot expiry info" err="Failed to begin transaction: context deadline exceeded"
[2026/09/09 07:19:26 EDT] incusd: time="2026-09-09T11:19:26Z" level=error msg="Failed getting custom volume info" err="Failed to begin transaction: context deadline exceeded"
[2026/09/09 07:20:20 EDT] incusd: time="2026-09-09T11:20:20Z" level=error msg="Failed getting instance snapshot expiry info" err="Failed to begin transaction: failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 07:20:26 EDT] incusd: time="2026-09-09T11:20:26Z" level=error msg="Failed getting custom volume info" err="Failed to begin transaction: context deadline exceeded"
[2026/09/09 07:20:31 EDT] incusd: time="2026-09-09T11:20:31Z" level=error msg="Failed getting custom volume info" err="Failed to begin transaction: failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 07:21:02 EDT] incusd: time="2026-09-09T11:21:02Z" level=error msg="Failed getting instance snapshot expiry info" err="Failed to begin transaction: failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 07:21:34 EDT] incusd: time="2026-09-09T11:21:34Z" level=error msg="Failed getting instance snapshot expiry info" err="Failed to begin transaction: failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 07:22:26 EDT] incusd: time="2026-09-09T11:22:26Z" level=error msg="Failed getting custom volume info" err="Failed to begin transaction: context deadline exceeded"
[2026/09/09 07:22:37 EDT] incusd: time="2026-09-09T11:22:37Z" level=error msg="Failed getting instance snapshot expiry info" err="Failed to begin transaction: failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 07:22:48 EDT] incusd: time="2026-09-09T11:22:48Z" level=error msg="Failed getting custom volume info" err="Failed to begin transaction: failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 07:23:42 EDT] incusd: time="2026-09-09T11:23:42Z" level=error msg="Failed getting instance snapshot expiry info" err="Failed to begin transaction: failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 07:24:03 EDT] incusd: time="2026-09-09T11:24:03Z" level=error msg="Failed getting custom volume info" err="Failed to begin transaction: failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 07:25:17 EDT] incusd: time="2026-09-09T11:25:17Z" level=error msg="Failed getting custom volume info" err="Failed to begin transaction: failed to create cowsql connection: no available cowsql leader server found"

and what I think is it trying to start the incus service

[2026/09/09 03:22:49 EDT] dnsmasq-dhcp: DHCP, sockets bound exclusively to interface incusbr0
[2026/09/09 03:22:49 EDT] dnsmasq: using only locally-known addresses for incus
[2026/09/09 03:22:49 EDT] dnsmasq: reading /etc/resolv.conf
[2026/09/09 03:22:49 EDT] dnsmasq: using nameserver 127.0.0.53#53
[2026/09/09 03:22:49 EDT] dnsmasq: using only locally-known addresses for incus
[2026/09/09 03:22:49 EDT] dnsmasq: read /etc/hosts - 7 names
[2026/09/09 03:23:02 EDT] incusd: time="2026-09-09T07:23:02Z" level=error msg="Failed creating leftover image clean up operation" err="failed to add "Pruning leftover image files" Operation dafc158f-d6bb-45c3-817c-e3b6436dc930 to database: Failed to begin transaction: failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 03:23:13 EDT] incusd: time="2026-09-09T07:23:13Z" level=error msg="Failed to start the daemon" err="Failed loading local instances: Failed to begin transaction: failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 03:23:48 EDT] incusd: time="2026-09-09T07:23:48Z" level=warning msg="Could not handover member's responsibilities" err="context deadline exceeded"
[2026/09/09 03:23:48 EDT] incusd: time="2026-09-09T07:23:48Z" level=warning msg="Loading local instances from disk as database is not available" err="Failed to begin transaction: sql: database is closed"
[2026/09/09 03:23:48 EDT] incusd: time="2026-09-09T07:23:48Z" level=warning msg="Failed loading instance" backup_file=/var/lib/incus/virtual-machines/incus-vm-win11-01/backup.yaml err="Failed to begin transaction: sql: database is closed" instance=incus-vm-win11-01 project=default
[2026/09/09 03:23:48 EDT] incusd: time="2026-09-09T07:23:48Z" level=warning msg="Failed loading instance from database to handle monitor event, trying backup file" err="Failed to begin transaction: sql: database is closed" instance=incus-vm-win11-01 project=default
[2026/09/09 03:23:48 EDT] incusd: time="2026-09-09T07:23:48Z" level=error msg="Failed loading instance to handle monitor event" err="Failed to begin transaction: sql: database is closed" instance=incus-vm-win11-01 project=default
[2026/09/09 03:23:48 EDT] incusd: Error: Failed loading local instances: Failed to begin transaction: failed to create cowsql connection: no available cowsql leader server found
[2026/09/09 03:23:48 EDT] systemd: incus.service: Main process exited, code=exited, status=1/FAILURE
[2026/09/09 03:32:40 EDT] incusd: Error: Daemon still not running after 600s timeout (Get "``http://unix.socket/1.0``": EOF)
[2026/09/09 03:32:40 EDT] systemd: incus.service: Control process exited, code=exited, status=1/FAILURE
[2026/09/09 03:32:40 EDT] systemd: incus.service: Failed with result 'exit-code'.
[2026/09/09 03:32:40 EDT] systemd: incus.service: Unit process 73644 (swtpm) remains running after unit stopped.
[2026/09/09 03:32:40 EDT] systemd: incus.service: Unit process 73674 (qemu-system-x86) remains running after unit stopped.
[2026/09/09 03:32:40 EDT] systemd: incus.service: Unit process 152639 (dnsmasq) remains running after unit stopped.
[2026/09/09 03:32:40 EDT] systemd: Failed to start incus.service - Incus - Daemon.
[2026/09/09 03:32:40 EDT] kernel: kauditd_printk_skb: 26 callbacks suppressed
[2026/09/09 03:32:40 EDT] kernel: audit: type=1130 audit(1788939160.885:658): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=incus comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[2026/09/09 03:32:40 EDT] systemd: incus.service: Consumed 29.357s CPU time, 28.7G memory peak.
[2026/09/09 03:32:41 EDT] systemd: incus.service: Scheduled restart job, restart counter is at 12.
[2026/09/09 03:32:41 EDT] systemd: incus.service: Found left-over process 73644 (swtpm) in control group while starting unit. Ignoring.
[2026/09/09 03:32:41 EDT] systemd: incus.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
[2026/09/09 03:32:41 EDT] systemd: incus.service: Found left-over process 73674 (qemu-system-x86) in control group while starting unit. Ignoring.
[2026/09/09 03:32:41 EDT] systemd: incus.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
[2026/09/09 03:32:41 EDT] systemd: incus.service: Found left-over process 152639 (dnsmasq) in control group while starting unit. Ignoring.
[2026/09/09 03:32:41 EDT] systemd: incus.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
[2026/09/09 03:32:41 EDT] systemd: Starting incus.service - Incus - Daemon...
[2026/09/09 03:32:41 EDT] (incusd): incus.service: Referenced but unset environment variable evaluates to an empty string: INCUS_OPTS
[2026/09/09 03:32:41 EDT] incusd: time="2026-09-09T07:32:41Z" level=warning msg="QMP monitor read failed" err="read unix @->/tmp/3146735805: use of closed network connection"
[2026/09/09 03:32:41 EDT] incusd: 2026/09/09 07:32:41 http: TLS handshake error from 192.168.2.45:64480: remote error: tls: unknown certificate
[2026/09/09 03:32:41 EDT] incusd: 2026/09/09 07:32:41 http: TLS handshake error from 192.168.2.45:64479: remote error: tls: unknown certificate
[2026/09/09 03:34:07 EDT] incusd: time="2026-09-09T07:34:07Z" level=error msg="Failed connecting to global database" attempt=6 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 03:34:19 EDT] incusd: time="2026-09-09T07:34:19Z" level=error msg="Failed connecting to global database" attempt=7 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 03:34:32 EDT] incusd: time="2026-09-09T07:34:32Z" level=error msg="Failed connecting to global database" attempt=8 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 03:34:44 EDT] incusd: time="2026-09-09T07:34:44Z" level=error msg="Failed connecting to global database" attempt=9 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 03:34:57 EDT] incusd: time="2026-09-09T07:34:57Z" level=error msg="Failed connecting to global database" attempt=10 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 03:35:10 EDT] incusd: time="2026-09-09T07:35:10Z" level=error msg="Failed connecting to global database" attempt=11 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/09 03:35:16 EDT] incusd: 2026/09/09 07:35:16 http: panic serving 10.10.10.101:52686: runtime error: invalid memory address or nil pointer dereference
[2026/09/09 03:35:16 EDT] incusd: goroutine 1906 [running]:
[2026/09/09 03:35:16 EDT] incusd: net/http.(*conn).serve.func1()
[2026/09/09 03:35:16 EDT] incusd: /usr/local/go/src/net/http/server.go:1939 +0xbb
[2026/09/09 03:35:16 EDT] incusd: panic({0x3dc8a78?, 0x407d730?})
[2026/09/09 03:35:16 EDT] incusd: /usr/local/go/src/runtime/panic.go:859 +0x125
[2026/09/09 03:35:16 EDT] incusd: ``github.com/lxc/incus/v7/internal/server/db.(*Cluster).Transaction``(0x0?, {0x40518d0?, 0x46cd140?}, 0x1a82c80?)
[2026/09/09 03:35:16 EDT] incusd: /build/incus/internal/server/db/db.go:302 +0x33
[2026/09/09 03:35:16 EDT] incusd: ``github.com/lxc/incus/v7/internal/server/cluster.newRolesChanges``(0x26e3c7b99b00, 0x26e3c72d48c0, {0x26e3c7a8f5f0, 0x3, 0x0?}, {0x0, 0x0, 0x0?})
[2026/09/09 03:35:16 EDT] incusd: /build/incus/internal/server/cluster/membership.go:1093 +0x98
[2026/09/09 03:35:16 EDT] incusd: ``github.com/lxc/incus/v7/internal/server/cluster.Handover``(0x26e3c7b99b00, 0x26e3c72d48c0, {0x26e3c740f218, 0x11})
[2026/09/09 03:35:16 EDT] incusd: /build/incus/internal/server/cluster/membership.go:1070 +0x30a
[2026/09/09 03:35:16 EDT] incusd: main.internalClusterPostHandover(0x26e3c6845d48, 0x26e3c6663a40)
[2026/09/09 03:35:16 EDT] incusd: /build/incus/cmd/incusd/api_cluster.go:2888 +0x34d
[2026/09/09 03:35:16 EDT] incusd: main.(*Daemon).createCmd.func1.3({0x40690e0?, 0x26e3c6d419c0?, 0x48?, 0x7b?})
[2026/09/09 03:35:16 EDT] incusd: /build/incus/cmd/incusd/daemon.go:904 +0x425
[2026/09/09 03:35:16 EDT] incusd: main.(*Daemon).createCmd.func1({0x404d0c0, 0x26e3c7354d20}, 0x26e3c6663900)
[2026/09/09 03:35:16 EDT] incusd: /build/incus/cmd/incusd/daemon.go:915 +0x16b0
[2026/09/09 03:35:16 EDT] incusd: net/http.HandlerFunc.ServeHTTP(0x26e3c6e4c480?, {0x404d0c0?, 0x26e3c7354d20?}, 0x0?)
[2026/09/09 03:35:16 EDT] incusd: /usr/local/go/src/net/http/server.go:2338 +0x29
[2026/09/09 03:35:16 EDT] incusd: net/http.(*ServeMux).ServeHTTP(0x26e3c7521ac8?, {0x404d0c0, 0x26e3c7354d20}, 0x26e3c6663900)
[2026/09/09 03:35:16 EDT] incusd: /usr/local/go/src/net/http/server.go:2903 +0x1cf
[2026/09/09 03:35:16 EDT] incusd: main.(*httpServer).ServeHTTP(0x26e3c6b55d70, {0x404d0c0, 0x26e3c7354d20}, 0x26e3c6663900)
[2026/09/09 03:35:16 EDT] incusd: /build/incus/cmd/incusd/api.go:431 +0x154
[2026/09/09 03:35:16 EDT] incusd: net/http.serverHandler.ServeHTTP({0x10?}, {0x404d0c0?, 0x26e3c7354d20?}, 0x1?)
[2026/09/09 03:35:16 EDT] incusd: /usr/local/go/src/net/http/server.go:3413 +0x8e
[2026/09/09 03:35:16 EDT] incusd: net/http.(*conn).serve(0x26e3c7b1e750, {0x4051ac8, 0x26e3c6cf9e90})
[2026/09/09 03:35:16 EDT] incusd: /usr/local/go/src/net/http/server.go:2137 +0x6dc
[2026/09/09 03:35:16 EDT] incusd: created by net/http.(*Server).Serve in goroutine 167
[2026/09/09 03:35:16 EDT] incusd: /usr/local/go/src/net/http/server.go:3581 +0x4fd
[2026/09/09 03:35:16 EDT] kernel: audit: type=1325 audit(1788939316.424:659): table=incus_test_d49dd97f-30be-4188-9b39-a8b025fbd9e5:149 family=2 entries=1 op=nft_register_table pid=154061 subj=unconfined comm="nft"
[2026/09/09 03:35:16 EDT] kernel: audit: type=1300 audit(1788939316.424:659): arch=c000003e syscall=46 success=yes exit=120 a0=3 a1=7fff64c15bf0 a2=0 a3=0 items=0 ppid=153665 pid=154061 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="nft" exe="/usr/sbin/nft" subj=unconfined key=(null)
[2026/09/09 03:35:16 EDT] kernel: audit: type=1327 audit(1788939316.424:659): proctitle=6E667400637265617465007461626C6500696E6375735F746573745F64343964643937662D333062652D343138382D396233392D613862303235666264396535
[2026/09/09 03:35:16 EDT] kernel: audit: type=1325 audit(1788939316.430:660): table=incus_test_d49dd97f-30be-4188-9b39-a8b025fbd9e5:150 family=2 entries=1 op=nft_unregister_table pid=154062 subj=unconfined comm="nft"
[2026/09/09 03:35:16 EDT] kernel: audit: type=1300 audit(1788939316.430:660): arch=c000003e syscall=46 success=yes exit=112 a0=3 a1=7ffc40e3a060 a2=0 a3=0 items=0 ppid=153665 pid=154062 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="nft" exe="/usr/sbin/nft" subj=unconfined key=(null)
[2026/09/09 03:35:16 EDT] kernel: audit: type=1327 audit(1788939316.430:660): proctitle=6E66740064656C657465007461626C6500696E6375735F746573745F64343964643937662D333062652D343138382D396233392D613862303235666264396535
[2026/09/09 03:35:16 EDT] incusd: time="2026-09-09T07:35:16Z" level=warning msg="Could not notify all nodes of database upgrade" err="failed to notify peer 10.10.10.101:8443: failed to notify node about completed upgrade: Patch "``https://10.10.10.101:8443/internal/database\``": Unable to connect to: 10.10.10.101:8443 ([dial tcp 10.10.10.101:8443: connect: c$
[2026/09/09 03:35:16 EDT] kernel: audit: type=1325 audit(1788939316.580:661): table=incus:151 family=1 entries=5 op=nft_unregister_chain pid=154078 subj=unconfined comm="nft"
[2026/09/09 03:35:16 EDT] kernel: audit: type=1300 audit(1788939316.580:661): arch=c000003e syscall=46 success=yes exit=144 a0=3 a1=7ffd5d3f3d10 a2=0 a3=0 items=0 ppid=153665 pid=154078 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="nft" exe="/usr/sbin/nft" subj=unconfined key=(null)
[2026/09/09 03:35:16 EDT] kernel: audit: type=1327 audit(1788939316.580:661): proctitle=6E667400666C75736800636861696E00696E657400696E637573006677642E696E637573627230003B0064656C65746500636861696E00696E657400696E637573006677642E696E637573627230
[2026/09/09 03:35:16 EDT] kernel: audit: type=1325 audit(1788939316.593:662): table=incus:152 family=1 entries=5 op=nft_unregister_chain pid=154079 subj=unconfined comm="nft"

anyhow I’m just going to reboot all 3 nodes and see what happens

3

Also the same problem with DELL PowerEdge servers. Check BIOS and storage updates. I had a pending critical BIOS update. After installing updates and updating secure boot certificates, I was able to successfully boot into IncusOS 202609082242.

5

I’ll look into the crash as that’s the main clue we’ve got in here so far.

6

The only available update I had was for IDRAC which didn’t fix the issue. I’m just gonna rebuild all 3.

7

I’ve got an Incus fix for the crash, but if that’s just an Incus race, rebooting the servers should get things back up.

8

Would be good to do some testing on that if you haven’t wiped them yet as I want to make sure there’s no other issue lingering.

9

Looks like it is a race. Sometimes IncusOS boots, sometimes not. Unfortunately I cannot start the fallback listener to get the logs. Probably I pasted TLS certificate incorrectly. Is there a way to manually test connection to fallback-listener when the server is operational to test if it is actually working?

10

Also booting the previous version for me does not get passed the screen “IncusOS is tarting…”.

11

I’ve got some planned work to fix that part of the fallback listener so we can have it run all the time too.

incusd/cluster: Reject internal rebalance/handover requests during st… by stgraber · Pull Request #3971 · lxc/incus · GitHub has the fix for the Incus crash itself though it’s definitely a race and not something that the latest update touched (it’s got the same Incus as last week’s).

12

Update:

The fallback server was working fine. The problem was that I tried to add it using incus remote add, and that required the Incus application running. After manually adding the address to ~/.config/incus/config.yml and downloading its TLS certificate to ~/.config/incus/servercerts/ I was able to interact with fallback-listener using incus admin os commands.

I downloaded the logs, but do not see any obvious errors:

[2026/09/10 11:47:50 CEST] incus-osd: 2026-09-10 11:47:50 INFO System is starting up machine-id=afb5287b4d114ce89dbc997824fa4f55 mode=production version=202609082242
[2026/09/10 11:48:56 CEST] incus-osd: 2026-09-10 11:48:56 INFO Starting application name=incus-ceph version=20.2.2 [202609082242]
[2026/09/10 11:48:56 CEST] incus-osd: 2026-09-10 11:48:56 INFO Starting application name=incus-linstor version=1.35.0 [202609082242]
[2026/09/10 11:48:56 CEST] incus-osd: 2026-09-10 11:48:56 INFO Starting application name=incus version=7.4 [202609082242]
...
[2026/09/10 11:48:56 CEST] systemd: Starting incus.socket - Incus - Daemon (unix socket)...
[2026/09/10 11:48:56 CEST] systemd: Listening on incus.socket - Incus - Daemon (unix socket).
[2026/09/10 11:48:56 CEST] systemd: Starting incus-lxcfs.service - Incus - LXCFS daemon...
[2026/09/10 11:48:56 CEST] systemd: Starting incus-startup.service - Incus - Startup check...
[2026/09/10 11:48:56 CEST] systemd: Started incus-lxcfs.service - Incus - LXCFS daemon.
[2026/09/10 11:48:56 CEST] audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=incus-lxcfs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2026/09/10 11:48:56 CEST] (lxcfs): incus-lxcfs.service: Referenced but unset environment variable evaluates to an empty string: LXCFS_OPTS
[2026/09/10 11:48:56 CEST] systemd: Starting incus.service - Incus - Daemon...
[2026/09/10 11:48:56 CEST] lxcfs: Starting LXCFS at /opt/incus/bin/lxcfs
[2026/09/10 11:48:56 CEST] (incusd): incus.service: Referenced but unset environment variable evaluates to an empty string: INCUS_OPTS
[2026/09/10 11:48:57 CEST] incusd: time="2026-09-10T11:48:57+02:00" level=warning msg="QMP monitor read failed" err="read unix @->/tmp/852717719: use of closed network connection"
[2026/09/10 11:50:22 CEST] incusd: time="2026-09-10T11:50:22+02:00" level=error msg="Failed connecting to global database" attempt=6 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 11:50:35 CEST] incusd: time="2026-09-10T11:50:35+02:00" level=error msg="Failed connecting to global database" attempt=7 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 11:50:47 CEST] incusd: time="2026-09-10T11:50:47+02:00" level=error msg="Failed connecting to global database" attempt=8 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 11:50:59 CEST] incusd: time="2026-09-10T11:50:59+02:00" level=error msg="Failed connecting to global database" attempt=9 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 11:51:12 CEST] incusd: time="2026-09-10T11:51:12+02:00" level=error msg="Failed connecting to global database" attempt=10 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 11:51:24 CEST] incusd: time="2026-09-10T11:51:24+02:00" level=error msg="Failed connecting to global database" attempt=11 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 11:51:37 CEST] incusd: time="2026-09-10T11:51:37+02:00" level=error msg="Failed connecting to global database" attempt=12 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 11:51:49 CEST] incusd: time="2026-09-10T11:51:49+02:00" level=error msg="Failed connecting to global database" attempt=13 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 11:52:01 CEST] incusd: time="2026-09-10T11:52:01+02:00" level=error msg="Failed connecting to global database" attempt=14 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 11:52:14 CEST] incusd: time="2026-09-10T11:52:14+02:00" level=error msg="Failed connecting to global database" attempt=15 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 11:53:16 CEST] incusd: time="2026-09-10T11:53:16+02:00" level=error msg="Failed connecting to global database" attempt=20 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 11:54:18 CEST] incusd: time="2026-09-10T11:54:18+02:00" level=error msg="Failed connecting to global database" attempt=25 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 11:55:20 CEST] incusd: time="2026-09-10T11:55:20+02:00" level=error msg="Failed connecting to global database" attempt=30 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 11:56:22 CEST] incusd: time="2026-09-10T11:56:22+02:00" level=error msg="Failed connecting to global database" attempt=35 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 11:57:24 CEST] incusd: time="2026-09-10T11:57:24+02:00" level=error msg="Failed connecting to global database" attempt=40 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 11:58:26 CEST] incusd: time="2026-09-10T11:58:26+02:00" level=error msg="Failed connecting to global database" attempt=45 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 11:58:56 CEST] systemd: incus-startup.service: start operation timed out. Terminating.
[2026/09/10 11:58:56 CEST] systemd: incus.service: start-post operation timed out. Terminating.
[2026/09/10 11:58:56 CEST] systemd: incus-startup.service: Main process exited, code=killed, status=15/TERM
[2026/09/10 11:58:56 CEST] systemd: incus-startup.service: Failed with result 'timeout'.
[2026/09/10 11:58:56 CEST] systemd: Failed to start incus-startup.service - Incus - Startup check.
[2026/09/10 11:58:56 CEST] audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=incus-startup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[2026/09/10 11:58:56 CEST] systemd: incus.service: Control process exited, code=killed, status=15/TERM
[2026/09/10 11:59:28 CEST] incusd: time="2026-09-10T11:59:28+02:00" level=error msg="Failed connecting to global database" attempt=50 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 12:00:30 CEST] incusd: time="2026-09-10T12:00:30+02:00" level=error msg="Failed connecting to global database" attempt=55 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 12:01:04 CEST] audit: SYSCALL arch=c000003e syscall=257 success=yes exit=5 a0=7 a1=5641afa11ad0 a2=2a0000 a3=0 items=1 ppid=1 pid=4261 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-tmpfile" exe="/usr/bin/systemd-tmpfiles" subj=unconfined key="incus-os-secrets"
[2026/09/10 12:01:04 CEST] audit: SYSCALL arch=c000003e syscall=257 success=yes exit=8 a0=ffffffffffffff9c a1=7fff3a488490 a2=280100 a3=0 items=1 ppid=1 pid=4261 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-tmpfile" exe="/usr/bin/systemd-tmpfiles" subj=unconfined key="incus-os-secrets"
[2026/09/10 12:01:32 CEST] incusd: time="2026-09-10T12:01:32+02:00" level=error msg="Failed connecting to global database" attempt=60 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 12:02:34 CEST] incusd: time="2026-09-10T12:02:34+02:00" level=error msg="Failed connecting to global database" attempt=65 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 12:03:36 CEST] incusd: time="2026-09-10T12:03:36+02:00" level=error msg="Failed connecting to global database" attempt=70 err="failed to create cowsql connection: no available cowsql leader server found"
[2026/09/10 12:04:26 CEST] systemd: incus.service: State 'stop-sigterm' timed out. Killing.
[2026/09/10 12:04:26 CEST] systemd: incus.service: Killing process 3236 (incusd) with signal SIGKILL.
[2026/09/10 12:04:26 CEST] systemd: incus.service: Main process exited, code=killed, status=9/KILL
[2026/09/10 12:04:26 CEST] systemd: incus.service: Failed with result 'timeout'.
[2026/09/10 12:04:26 CEST] systemd: Failed to start incus.service - Incus - Daemon.
[2026/09/10 12:04:26 CEST] audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=incus comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[2026/09/10 12:04:26 CEST] systemd: incus.service: Consumed 3.825s CPU time, 270.6M memory peak.
[2026/09/10 12:04:26 CEST] incus-osd: 2026-09-10 12:04:26 ERROR Failed to run: systemctl start incus.socket incus-lxcfs.service incus-startup.service incus.service: exit status 1 (Job for incus-startup.service failed because a timeout was exceeded.
[2026/09/10 12:04:26 CEST] incus-osd: See "systemctl status incus-startup.service" and "journalctl -xeu incus-startup.service" for details.
[2026/09/10 12:04:26 CEST] incus-osd: Job for incus.service failed because a timeout was exceeded.
[2026/09/10 12:04:26 CEST] incus-osd: See "systemctl status incus.service" and "journalctl -xeu incus.service" for details.)
[2026/09/10 12:04:26 CEST] incus-osd: 2026-09-10 12:04:26 WARN Primary application incus failed to start; attempting to enable fallback HTTPS server for basic connectivity
...
[2026/09/10 12:04:26 CEST] incus-osd: 2026-09-10 12:04:26 INFO System is ready version=202609082242

If necessary I can provide the full log.

I noticed that there was an update 202609100026 avaiable and tried to install it using incus admin os system update check. That took some time. Looks like IncusOS was trying to restart the failing incusd service and download of updates took only between these restarts (each with 10 min timeout).

After the restart, the system lacked primary Incus application:

Screenshot 2026-09-10 at 13.54.15
Screenshot 2026-09-10 at 13.54.152364Ă—1772 334 KB

After booting back to version 202609082242 the primary application was missing as well. Possibly the update 202609100026 was pulled back in the middle of the process (because it took very long).

Screenshot 2026-09-10 at 14.01.29
Screenshot 2026-09-10 at 14.01.291920Ă—1437 369 KB

Other cluster members managed to update to 202609100026 but similarly failed to start incusd with the same errors failed to create cowsql connection: no available cowsql leader server found.

13

@gibmat any idea what’s going on with the missing primary application?

I suspect we’re going to need to look at the state file as somehow having incus-ceph and incus-linstor but no incus is very unexpected…

@ykazakov any chance you can boot into a live environment and use the recovery key to unlock the root disk and then pull /var/lib/incus-os/state.txt?

14

So I re-installed (got secureboot working this time. yay) and the same thing happened to me. But - it worked initially.

I think the iso puts version 202609030314 on . And then I get upgraded to 202609100026.

You’d think that when I reboot I could pick 202609030314 and then it’d work but that version just hangs at IncusOS is starting…

15

Here I pulled this file. I redacted some sensitive information.

#Version: 10
SecureBoot.Version: 202609100026
SecureBoot.FullyApplied: true
ServicesSeedApplied: true
Applications.Incus.State.Initialized: true
Applications.Incus.State.FriendlyVersion: 7.4 [202609100026]
Applications.Incus.State.Version: 202609100026
Applications.IncusCeph.State.Initialized: true
Applications.IncusCeph.State.FriendlyVersion: 20.2.2 [202609082242]
Applications.IncusCeph.State.Version: 202609082242
Applications.IncusCeph.State.AvailableVersions[0]: 202609082242
Applications.IncusCeph.State.AvailableVersions[1]: 202609100026
Applications.IncusLinstor.State.Initialized: true
Applications.IncusLinstor.State.FriendlyVersion: 1.35.0 [202609082242]
Applications.IncusLinstor.State.Version: 202609082242
Applications.IncusLinstor.State.AvailableVersions[0]: 202609082242
Applications.IncusLinstor.State.AvailableVersions[1]: 202609100026
OS.Name: IncusOS
OS.RunningRelease: 202609082242
OS.NextRelease: 202609100026
OS.SuccessfulBoot: true
Services.Ceph.Config.Enabled: true
Services.Ceph.Config.Clusters[ceph].FSID: b8ac9bba-a431-4ac0-88c2-4c64c4857a62
Services.Ceph.Config.Clusters[ceph].Monitors[0]: 10.40.0.21
Services.Ceph.Config.Clusters[ceph].Monitors[1]: 10.40.0.22
Services.Ceph.Config.Clusters[ceph].Monitors[2]: 10.40.0.23
Services.Ceph.Config.Clusters[ceph].Keyrings[admin].Key: XXXXXXXXXXXXXXXXXXXX==
Services.Ceph.Config.Clusters[ceph].ClientConfig[mon_host]: [v2:10.40.0.21:3300,v1:10.40.0.21:6789],[v2:10.40.0.22:3300,v1:10.40.0.22:6789],[v2:10.40.0.23:3300,v1:10.40.0.23:6789]
Services.Ceph.Config.Clusters[ceph].ClientConfig[ms_bind_msgr1]: true
Services.Ceph.Config.Clusters[ceph].ClientConfig[ms_bind_msgr2]: true
Services.LVM.Config.Enabled: true
Services.OVN.Config.Enabled: true
Services.OVN.Config.Database: ssl:10.50.0.61:6642,ssl:10.50.0.62:6642,ssl:10.50.0.63:6642
Services.OVN.Config.TLSClientCertificate: -----BEGIN CERTIFICATE-----\nXXXXXXXXXXXXXX\n-----END CERTIFICATE-----
Services.OVN.Config.TLSClientKey: -----BEGIN PRIVATE KEY-----\nXXXXXXXXXXXXXX\n-----END PRIVATE KEY-----
Services.OVN.Config.TLSCACertificate: -----BEGIN CERTIFICATE-----\nXXXXXXXXXXXXXX\n-----END CERTIFICATE-----
Services.OVN.Config.TunnelAddress: 10.50.0.13
Services.OVN.Config.TunnelProtocol: geneve
System.FallbackListener.Config.ListenAddress: 0.0.0.0:9443
System.FallbackListener.Config.TrustedClientCertificates[0]: -----BEGIN CERTIFICATE-----\nXXXXXXXXXXXXXX\n-----END CERTIFICATE-----\n
System.Network.Config.DNS.Nameservers[0]: xxx.xx.x.xxx
System.Network.Config.DNS.Nameservers[1]: xxx.xx.xxx.xxx
System.Network.Config.DNS.Nameservers[2]: xxxx:xxx:xxxx::xxx
System.Network.Config.DNS.Nameservers[3]: xxxx:xxx:xxxx:x::xxx
System.Network.Config.DNS.SearchDomains[0]: my-domain.org
System.Network.Config.Time.NTPServers[0]: ntp1.my-domain.org
System.Network.Config.Time.NTPServers[1]: ntp2.my-domain.org
System.Network.Config.Time.NTPServers[2]: ntp2.my-domain.org
System.Network.Config.Time.Timezone: Europe/Berlin
System.Network.Config.Interfaces[0].Addresses[0]: xxx.xx.xx.xxx/24
System.Network.Config.Interfaces[0].Addresses[1]: xxxx:xxx:xxxx:xxx::xxx/64
System.Network.Config.Interfaces[0].Hwaddr: xx:xx:xx:xx:xx:xx
System.Network.Config.Interfaces[0].Name: xyz
System.Network.Config.Interfaces[0].Roles[0]: management
System.Network.Config.Interfaces[0].Routes[0].To: 0.0.0.0/0
System.Network.Config.Interfaces[0].Routes[0].Via: xxx.xx.xx.1
System.Network.Config.Interfaces[0].Routes[1].To: ::/0
System.Network.Config.Interfaces[0].Routes[1].Via: xxxx:xxx:xxxx:xxx::1
System.Network.Config.Interfaces[1].Hwaddr: xx:xx:xx:xx:xx:xx
System.Network.Config.Interfaces[1].MTU: 9000
System.Network.Config.Interfaces[1].Name: switch
System.Network.Config.Interfaces[1].RequiredForOnline: no
System.Network.Config.Interfaces[1].Roles[0]: instances
System.Network.Config.Interfaces[1].VLANTags[0]: 10
System.Network.Config.Interfaces[1].VLANTags[1]: 20
System.Network.Config.Interfaces[1].VLANTags[2]: 30
System.Network.Config.Interfaces[1].VLANTags[3]: 40
System.Network.Config.Interfaces[1].VLANTags[4]: 50
System.Network.Config.VLANs[0].Addresses[0]: 10.10.0.13/24
System.Network.Config.VLANs[0].ID: 10
System.Network.Config.VLANs[0].Name: incus
System.Network.Config.VLANs[0].Parent: switch
System.Network.Config.VLANs[0].Roles[0]: cluster
System.Network.Config.VLANs[1].Addresses[0]: 10.20.0.13/24
System.Network.Config.VLANs[1].ID: 20
System.Network.Config.VLANs[1].Name: linstor
System.Network.Config.VLANs[1].Parent: switch
System.Network.Config.VLANs[2].Addresses[0]: 10.30.0.13/24
System.Network.Config.VLANs[2].ID: 30
System.Network.Config.VLANs[2].Name: drbd
System.Network.Config.VLANs[2].Parent: switch
System.Network.Config.VLANs[3].Addresses[0]: 10.40.0.13/24
System.Network.Config.VLANs[3].ID: 40
System.Network.Config.VLANs[3].Name: ceph
System.Network.Config.VLANs[3].Parent: switch
System.Network.Config.VLANs[4].Addresses[0]: 10.50.0.13/24
System.Network.Config.VLANs[4].ID: 50
System.Network.Config.VLANs[4].Name: ovn
System.Network.Config.VLANs[4].Parent: switch
System.Provider.Config.Name: operations-center
System.Provider.Config.Config[server_certificate]: -----BEGIN CERTIFICATE-----\nnXXXXXXXXXXXXXX\n-----END CERTIFICATE-----\n\n-----BEGIN CERTIFICATE-----\nnXXXXXXXXXXXXXX\n-----END CERTIFICATE-----\n\n-----BEGIN CERTIFICATE-----\nnXXXXXXXXXXXXXX\n-----END CERTIFICATE-----\n\n-----BEGIN CERTIFICATE-----\nnXXXXXXXXXXXXXX\n-----END CERTIFICATE-----\n\n-----BEGIN CERTIFICATE-----\nnXXXXXXXXXXXXXX\n-----END CERTIFICATE-----\n\n-----BEGIN CERTIFICATE-----\nnXXXXXXXXXXXXXX\n-----END CERTIFICATE-----\n\n-----BEGIN CERTIFICATE-----\nnXXXXXXXXXXXXXX\n-----END CERTIFICATE-----\n
System.Provider.Config.Config[server_token]: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
System.Provider.Config.Config[server_url]: https://operations-center.my-domain.org:8443
System.Provider.State.Registered: true
System.Security.Config.EncryptionRecoveryKeys[0]: xxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx
System.Security.State.EncryptionRecoveryKeysRetrieved: true
System.Update.Config.Channel: stable
System.Update.Config.CheckFrequency: never
System.Storage.Config.ScrubSchedule: 0 4 * * 0
System.Storage.Config.TrimSchedule: 0 4 * * 6

Is there a way to add the Incus application back to this server?

16

@gibmat can you look into this one please?

17

@ykazakov From the live environment with the root partition unlocked, can you also show the output of ls /var/lib/incus-os-extentions/* ?

From the state file it appears that your IncusOS system isn’t finding any actual available installed versions of the Incus application, even though it’s present in the state. I’m curious to see if any incus.raw files exist on disk, or if they’ve somehow all disappeared.

The good news is that I don’t think this will be too hard to recover from. I can add some logic to how IncusOS handles applications to trigger a re-download of the application if it’s present and marked as initialized in the state but missing from the disk.

18

Looks like there is no incus.raw:

root@ubuntu-server:/mnt/incusos# ls var/lib/incus-os-extensions/*
var/lib/incus-os-extensions/202609082242:
incus-ceph.raw  incus-linstor.raw

var/lib/incus-os-extensions/202609100026:
incus-ceph.raw  incus-linstor.raw

Is it also possible to recover from the cowsql database error?

2026/09/10 11:50:22 CEST] incusd: time=“2026-09-10T11:50:22+02:00” level=error msg=“Failed connecting to global database” attempt=6 err=“failed to create cowsql connection: no available cowsql leader server found”

If not, I can just wipe everything and start over. This was just a test cluster with no important data.

19

The no available cowsql leader server found usually just means that you don’t have a quorum of Incus servers running. You need at least 2 out of 3 servers with the database role to be online for a leader to get elected.

20

So it is not possible to start a cluster if all members are down?