IncusOS unable to boot without network

IncusOS
1

I run opnsense as a vm to use excess resources for hosting light services, this means until the vm has booted there is no network, but when booting IncusOS with this setup i get the following error:

Error Failed to check for Secure Boot Key updates: http request timed out after five seconds: Get “https://images.linuxcontainers.org/os/index.sjson”: dial tcp: lookup images.linuxcontainers.org on 127.0.0.53:53: server misbehaving (provider: images)

if i reboot with regular non-vm hosted network it boots fine, and once booted the vm also starts without issue.

also: what is the correct method for nic passthrough on IncusOS, i added the following to opnsense yaml and it works but i’m unsure if it’s adding an extra layer between the nic and vm. on proxmox you would pass all functions of the nic pci id

devices:
    nic-0:
        nictype: physical
        parent: enp1s0f0
        type: nic

    nic-1:
        nictype: physical
        parent: enp1s0f1
        type: nic

    nic-2:
        nictype: physical
        parent: enp1s0f2
        type: nic

    nic-3:
        nictype: physical
        parent: enp1s0f3
        type: nic
2

Addressing the first part of your post, are you running opnsense as a VM hosted on IncusOS? I know a few people have experimented with that sort of a setup; this might be a good opportunity to improve that particular use case. We have support for a similar setup when IncusOS self-hosts Operations Center, but nothing generic to indicate that IncusOS should delay update checks for a specific duration or until some network endpoint becomes reachable.

The error you’re seeing is an update check error, which shouldn’t actually prevent IncusOS from starting up and becoming available. I tested booting one of my physical IncusOS machines with its ethernet cable unplugged; I see a similar error and then further message as Incus starts and IncusOS completes startup.

If routing for network access to IncusOS requires going through your opnsense VM, you may want to enable boot.autostart in the VM’s configuration (see Instance options - Incus documentation ).

3

Hallo :wave:
I’d like to jump in since I am facing the same issue and also since that post is not that old :slight_smile:
I understand that there is a will to improve this type of scenario on Incus side and I will follow the progress.
In the mean time, I’d like to make sure that the two things that seems to hang on boot are readressed once there is internet connectivity, which are:

  • NTP synchronization
  • Secure Boot keys updates

I can work with my system coming online in 3 minutes instead of 30 seconds but I want to make sure that does not create a degraded system or if I need to take any measure to avoid that.

(FWIW, network interfaces are not using DHCP nor SLAAC, they have static IPs)

os_version: "202607060039"

Cheers !

4

Both of those will re-trigger on their own later.

For NTP, we have systemd-timesyncd running even if it fails to sync initially, so it won’t be long before it detects NTP access and will correct any time synchronization issues.

For updates (both SecureBoot and regular OS checks), unless the check frequency was set to never, it will typically recheck within 6 hours.

1 Like
5

Is it possible that if I set

config:
  syslog:
    address: log.server:514

but since DNS resolution does not work on boot time, systemd-netlogd.service never get updated so even once DNS resolution is working, no logs flowing through until a restart of systemd-netlogd.service ?

6

That’s definitely possible. systemd-netlogd works well enough but it’s not the most polished of the systemd components. I’d strongly recommend using an IP address :slight_smile:

7

Thanks @stgraber

[After debugging, not thread related]

So I went ahead and let it do its work (update during a maintenance window) and it looks like at the same time, a new version of IncusOS was released so it seems to have updated it as well. Since it is configured to auto_reboot on update, it rebooted onto the new version, no network and now it is throwing me the following error, with or without network…

ERROR Failed to run: systemd-sysext refresh: exist status 1 (Failed to read metadatafor image incus: Structure needs cleaning)

Relevant log lines at the update timestamp

2026-07-20 06:01:26 INFO Downloading application update application=incus channel=stable version=202607192224
DMAR: DRHD: handling fault status reg 3
DMAR: [DMA Read NO_PASID] Request device [00:1f.2] fault addr 0xd6400000 [fault reason 0x0c] non-zero reserved fields in PTE
ata1.00: exception Emask 0x70 SAct 0x1ff60 SErr 0x440900 action 0x6 frozen
ata1.00: irq_stat 0x28000000, host bus error, interface fatal error
ata1: SError: { UnrecovData HostInt CommWake Handshk }
ata1.00: failed command: WRITE FPDMA QUEUED
ata1.00: cmd 61/00:10:00:6d:35/20:00:01:00:00/40 tag 2 ncq dma 4194304 ou
         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x70 (host bus error)
ata1.00: status: { DRDY }
ata1: hard resetting link
ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata1.00: configured for UDMA/133
ata1: EH complete

Trying to boot on previous version but getting the same problem.

  • SSD (3 months OLD) passed all checks
  • RAM passed 5+ memtest86+

I do agree that it seems to point to a hardware related issue but every tests on that said hardware returned SUCCESS without any error and the update maintenance windows worked well in the past (before trying it without network/internet access on reboot)

Questions

  1. Is this related ? (I do not want to hijack a thread if it is not)
  2. Is it expected to happen in the case an update is applied then the system reboot and have no internet connectivity ?

Update:
So, FWIW; does not seems to be network related

Altough, and this is where i’d like @stgraber or @gibmat input:
After some debugging, using IncusOS 202607060039 and deleting the Incus application 202607192224 forced a new download of the application and it worked just fine.

Same thing didn’t worked on IncusOS 202607192224 so it seems to me like something on the OS side (kernel ?) changed in regards of SATA support.

Scenario 1: IncusOS 202607192224 with Incus 202607192224 → X
Scenario 2: IncusOS 202607060039 with Incus 202607192224 → works

I did try the scenario 1 a couple of time by forcing the redownload of the image and always getting a corrupted image (verified through gdisk)

Might worth adding, a incus 202607192224 downloaded through IncusOS 202607060039 works on IncusOS 202607192224 which seems to lead to latest OS version is not dealing well with (my) SATA controller and therefore, corrupt the image ?