IncusOS booted and says “System is ready”, but remote Incus API refuses connections after power-off

1

I’m running IncusOS on a homelab server. It used to work, but after powering off the server and booting it again, the Incus client can no longer connect.

The machine appears to boot successfully. On the attached display/logs I see Incus start normally, including: “System is ready” together with the Incus version. The server network interface eno4 has: IPv4: 10.155.0.23, IPv6: 2607:f598:dc00:767:3eec:efff:fe78:23bf.

From my Mac/client machine, the host is reachable: ping 10.155.0.23 works with no packet loss. But Incus cannot connect: incus list fails with: Error: Get "https://homelab-incus-server:8443/1.0": Unable to connect to: homelab-incus-server:8443 ([dial tcp 10.155.0.23:8443: connect: connection refused]). The client remote is configured as: homelab -> https://homelab-incus-server:8443 and /etc/hosts on the client maps: 10.155.0.23 homelab-incus-server.

I checked TCP reachability from the client: nc -vz 10.155.0.23 8443 returns: Connection refused. I also tested the IPv6 address: nc -vz 2607:f598:dc00:767:3eec:efff:fe78:23bf 8443 also returns: Connection refused.

SSH is not available, which I understand is expected for IncusOS: nc -vz 10.155.0.23 22 returns: Connection refused. Port 443 timed out.

So the host has network connectivity, but nothing appears to be accepting connections on the Incus API port. This seems to happen even though the IncusOS display/logs say the system is ready.

My suspicion is that IncusOS/Incus is running locally but the HTTPS listener is not actually bound to 10.155.0.23:8443 or [2607:f598:dc00:767:3eec:efff:fe78:23bf]:8443. Because this is IncusOS, I don’t have shell access to run ss, inspect core.https_address, or restart services manually.

Questions:

  1. What can cause IncusOS to show “System is ready” while the Incus API on :8443 is not listening?

  2. Is there a supported recovery path when the IncusOS API itself is unreachable?

  3. Should a fallback API endpoint appear in this situation? If so, where should I look for the port?

  4. Is there a way to recover or reset the Incus API listener configuration without reinstalling?

I’m attaching a screenshot of the current logs.

I tried reverting to a previous backup version, but it asked for some recovery keys and I got scared.

I also had to reboot several times, the first time I booted up the instance after the initial accidental power off had an error that said “Failed to check for secure boot key updates” or something along those lines. Rebooting made that error go away.

IncusOS version: `202605272150`

Screenshot 2026-05-28 at 12.58.08 AM
Screenshot 2026-05-28 at 12.58.08 AM1726×1246 171 KB

2

We do have a recently added mechanism to have IncusOS itself directly listen on the network too, providing a way to get logs, update applications and some other actions.

But that’s obviously not super useful when you can’t turn it on :wink:

@gibmat I don’t recall, do we automatically spawn it in the recovery boot scenario or is it always opt-in?

3

I’m happy to try it if we can turn it on. Otherwise, what are my options? Using the previous version?

Also wondering if you see any red flags in the logs. I think the warning regarding update channel is also suspicious.

4

An immediate red flag is that no applications are listed between “Bringing up the local storage” and “System is ready”…. Without a primary application there won’t be any API for a client to connect to.

What was the prior version of IncusOS that was installed? (You can see this in the boot menu options if you reboot the system.) Your mentioning the need for a recovery key hints that it might have been a while since the system was updated, as that should only happen if a TPM binding update was applied, and the last of those was back in mid-March if I remember correctly.

We do have a fallback HTTPS listener that can activate, but it requires a primary application to have been installed and initialized so it can use the same HTTPS certificates. But since no primary application failed to start, it wouldn’t be activated.

The “System is ready” message merely indicates that IncusOS has completed its startup tasks: networking, storage, services, applications, etc. It doesn’t indicate if a specific application is running without issue, since IncusOS can’t really know that.

When booted from the recovery/older version of IncusOS that’s installed, we do attempt to start the fallback HTTPS listener ( incus-os/incus-osd/cmd/incus-osd/main.go at main · lxc/incus-os · GitHub ) regardless of the primary application state. That doesn’t appear to be the case here, since I don’t see any log message about booting from a backup version of IncusOS.

That’s because you’re running a version of IncusOS (202605272150) that’s currently only available in the “testing” channel, but the system is configured to look for updates from the default “stable” channel.

5

I just tried to boot up from the previous version from February 23rd by introducing the key, and I’m getting the following error.

Screenshot 2026-05-28 at 4.53.16 PM
Screenshot 2026-05-28 at 4.53.16 PM1568×1186 307 KB

This is very unexpected.

Please, tell me there’s a way to restore the previous config and applications :sweat_smile:

6

Alright, this could be an adventure… there were at least three major updates to IncusOS since your prior installed version:

The panic you see when booting into the prior installed version is due to an update to the internal state format. When the new version of IncusOS booted, the state format was updated, and booting back into the old version is encountering a bug that’s since been fixed ( incus-osd/state: Guard against index out of range panic when decoding state by gibmat · Pull Request #1102 · lxc/incus-os · GitHub ).

My first suggestion is to reboot into the current version and grab the latest update of IncusOS (202605281810) that fixes a different set of bugs. I doubt it will automatically re-fetch the Incus application, but at least the OS will be updated.

Then, you’ll need to follow the steps outlined in Emergency Procedure for a Lost Client Certificate - IncusOS documentation to use your encryption recovery key to manually decrypt the IncusOS root partition.

The first order of business will be grabbing the state file: cat /mnt/incusos/var/lib/incus-os/state.txt (Note that the state file does contain secrets, so either redact fields, or DM it to me directly.)

Next, see what (if any) applications are present:

ls -ltr /mnt/incusos/var/lib/extensions/
ls -ltr /mnt/incusos/var/lib/incus-os-extensions/*

I’m hoping your state file doesn’t show any issues, and that we can manually download and symlink the Incus application.

(Since this will be a multi-step process, it would be helpful if you could keep the IncusOS system decrypted until we get everything fixed up.)

7

Same issue as @jvican here on IncusOS version 202605311846. Reboot reason was a power outage. The start screen looks the same, minus the WARN line. The API is not reachable and fails with connection refused. Booting into an older version 202603311955 fails with (formatted as is):

1xc/incus-os/incus-osd/internal/state.decodeHelper({0xd38080?, 0x34c2fa442608?,
3xe50f60?}, {0x34c2fa331900, 0x4, 0x4), (0x34c2fa45e571, 0x4))
Jun 01 12:19:34 03000200-0400-0500-0006-000700080009 incus-osd[809]: /ac
tions-runner/_work/incus-os/incus-os/incus-osd/internal/state/decode.go:123 +0x8
сб
Jun 01 12:19:34 0300020010400-0500-0006-000700080009 incus-osd[809]: ``github.com/
1xc/incus-os/incus-osd/internal/state.Decode({0x34c2fa45e000?, 0x1?, Øx5?}, {0x0
, 0x0, 0x34c2fa430dØf?},0x34c2fa442608)
Jun 01 12:19:34 03000200-0400-0500-0006-000700080009 incus-osd[809]: /ac
tions-runner/_work/incus-os/incus-os/incus-osd/internal/state/decode.go:68 +0x35
Jun 01 12:19:34 83099200-0480-0560-0006-000700080009 incus-osd18091:github.com/
1xc/incus-os/incus-osd/internal/state.LoadOrCreate((0x34c2fa3612a0, (0x1b})
Jun 01 12:19:34 03000200-0400-0500-0006-000700080009 incus-osd[8091: /ac
tions-runner/_work/incus-os/incus-os/incus-osd/internal/state/file.go:35 +0x23a
Jun 01 12:19:34 03000200-0400-0500-0006-000700080009 incus-osd[809]: main.main()
Jun 01 12:19:34 03000200-0400-0500-0006-000700080009 incus-osd[809]: /ac
tions-runner/_work/incus-os/incus-os/incus-osd/cd/incus-osd/main.go:71 +0x38b
Jun 01 12:19:34 03000200-0400-0500-0006-000700080009 systemd[1]: incus-osd.servi
ce: Main process exited, code-exited, status=2/INVALIDARGUMENT
Jun 01 12:19:34 03000200-0400-0500-0006-000700080009 systemd[1]: incus-osd.servi
ce: Failed with result ‘exit-code’
Jun 01 12:19:34 03000200-0400-0500-0006-000700080009 systemd[1]: incus-osd.servi
ce: Triggering OnFailure= dependencies.

How can we “grab the latest update of IncusOS” @gibmat ? So far I see no way to interact with the system. Happy to supply more log/triage if needed.

8

@jvican @ole.mn I was able to reproduce this issue locally and develop a fix. Details are available at Details on this week's IncusOS updates and steps for updating to the current stable release , and is fixed by Fix legacy system updates by gibmat · Pull Request #1142 · lxc/incus-os · GitHub .

IncusOS 202606030722 includes the fix; after rebooting your system it should be able to automatically fetch the update, apply it, reboot and you’ll be back to a fully functional system.

Edited 6/4 with stable IncusOS version that includes the fix.

1 Like
9

This is awesome. Was out for a roadtrip and just came back. I’m super happy to hear there’s a potential bugfix on the way. Thank you so much for your prompt follow-up @gibmat!

Do we know when the fix will roll out in the next stable IncusOS release? Is that daily?

By the way, I did happen to reboot my computer and it picked up one of the most recent stable versions. I got a different error, which I’m sharing just in case it’s useful for you at all.

Screenshot 2026-06-01 at 10.48.22 PM
Screenshot 2026-06-01 at 10.48.22 PM1868×1394 277 KB

Will update to the next stable IncusOS release as soon as you tell me it’s ready.

10

In this specific case, that’s an expected error. What’s happened is your IncusOS system was old enough that it hadn’t yet migrated to supporting versioned applications, so the systemd-sysext images under /var/lib/extensions/ aren’t symlinks as the current IncusOS logic expects. The next stable release will have updated logic to handle this migration edge case, which will then return your system to normal.

A new IncusOS build hasn’t rolled out yet, but should in the next day or two most likely.

11

Yeah, waiting for the 7.0.11 kernel to complete validation, then we’ll be pushing an update with it.

12

I’ve just updated to the 20260603 version and everything is now working!

Thank you so much for the fast and effective fix. :slight_smile:

Screenshot 2026-06-03 at 9.49.07 AM
Screenshot 2026-06-03 at 9.49.07 AM1568×1190 193 KB

13

Be careful as this is in the testing channel and we’d generally not recommend anyone run production systems from that channel :slight_smile:

That said, this image looks good and will likely be in stable later today.

14

Ah, interesting. My whole system is on the incus stable channel. How come I picked up an image in the testing channel?

 ❯ incus admin os system update show

WARNING: The IncusOS API and configuration is subject to change

config:
  auto_reboot: false
  channel: stable
  check_frequency: 6h
state:
  last_check: "2026-06-04T04:38:58.489035411Z"
  needs_reboot: true
  status: IncusOS has been updated to version 202606030722
15

Hmm, that’s definitely pretty odd…

@gibmat did we break something that would affect channel filtering?

16

Hmm, this is the second report in the past week that I’ve seen of an IncusOS system on the stable channel fetching an update from testing instead. I don’t think we’ve changed anything related to selecting the update channel recently, but will see if I can reproduce the issue locally.