IncusOS 7.0 - Errors after upgrading (corrupted blocks and systemd-sysext refresh)

IncusOS
1

Hallo :waving_hand:

First thing first, congrats on that new major release !

One of my IncusOS server was updated to 7.0 and my containers are no longer able to start since the network interface tied to them is no longer available

Error in UI:

Loading network state failed
Network interface "monitoring-br0" not found

Error in logs:
context-err="Failed starting: The DNS and DHCP service exited prematurely: exit status 127 (\"dnsmasq: error while loading shared libraries: /lib/x86_64-linux-gnu/libnetfilter_conntrack.so.3: cannot read file data: Input/output error\")" context-network="monitoring-br0" context-project="default" level="error" Failed initializing network

I was about to recreate the network interface but want to check if there is a solution since some of my servers have a lot of network interfaces and that might become a bigger problem :slight_smile:

Cheers !

2

Input/output error isn’t usually a great sign. I’d recommend you try to reboot that server.

Otherwise, look at incus admin os debug log for other errors.

3

Yeah, already tried the reboot but still the same.
Having a deeper look at the debug logs seems to indicate corrupted data block(s)… Weird.

Everything was just fine before reboot for upgrading… oh well

I guess i’ll have to restart this one from scratch.

4

@gibmat thoughts?

Do we need a way to reinstall an application to handle potential corruption of the sysext?

@dwlfrth we’ll have another stable image later today, so you could wait for that and see if updating up a newer version fixes the issue.

1 Like
5

Excellent, will wait :slight_smile:

6

I tried on another staging server and same thing. What are the odds of two corrupted blocks on two different systems that were working just fine before the upgrade ?

Not pointing finger, just want to understand if something is wrong on my end :slight_smile:

7

Not particularly likely :slight_smile:
What are you seeing in the logs?

8

Yeah, two different installs failing the same way pretty much rules out a hardware issue. :slightly_frowning_face:

What is the version of IncusOS that you’ve updated to?

We do have logic in place whenever the sysext images are refreshed to warn and remove any that are corrupt or otherwise don’t match their expected signature. This happens each time the system boots, as well as when applying updates. If @dwlfrth hasn’t seen any messages about “corrupt on-disk image, attempting to cleanup”, then the sysext images are at least good enough for systemd-sysext to load them.

9

There was that line before removing the NIC

[2026/05/07 15:58:58 EDT] kernel: device-mapper: verity: 259:3: data block 141651 is corrupted

And then

[2026/05/07 16:09:59 EDT] incusd: panic: runtime error: invalid memory address or nil pointer dereference
[2026/05/07 16:09:59 EDT] incusd: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1bed3ae]
[2026/05/07 16:09:59 EDT] incusd: goroutine 42502 [running]:
[2026/05/07 16:09:59 EDT] incusd: github.com/lxc/incus/v7/internal/server/device.(*unixCommon).validateConfig.func1({0xd3b6de6e9a0, 0x10})
[2026/05/07 16:09:59 EDT] incusd: 	/build/incus/internal/server/device/unix_common.go:59 +0x4e
[2026/05/07 16:09:59 EDT] incusd: github.com/lxc/incus/v7/internal/server/device/config.Device.Validate(0xd3b6d9ca420, 0xd3b6da58280)
[2026/05/07 16:09:59 EDT] incusd: 	/build/incus/internal/server/device/config/devices.go:26 +0x162
[2026/05/07 16:09:59 EDT] incusd: github.com/lxc/incus/v7/internal/server/device.(*unixCommon).validateConfig(0xd3b6de0bd10, {0x7ff7d83f8f80?, 0xd3b6dce8300?}, 0x0?)
[2026/05/07 16:09:59 EDT] incusd: 	/build/incus/internal/server/device/unix_common.go:122 +0x38b
[2026/05/07 16:09:59 EDT] incusd: github.com/lxc/incus/v7/internal/server/device.New({0x2a3bdd0, 0xd3b6dce8300}, 0xd3b6da00200, {0xd3b6de6e960, 0xa}, 0xd3b6d9ca420, 0x0, 0xd3b6cfbc660, 0xd3b6cfbc680)
[2026/05/07 16:09:59 EDT] incusd: 	/build/incus/internal/server/device/device_load.go:138 +0x1db
[2026/05/07 16:09:59 EDT] incusd: github.com/lxc/incus/v7/internal/server/instance/drivers.(*common).deviceLoad(0xd3b6dce8300, {0x2a3bdd0, 0xd3b6dce8300}, {0xd3b6de6e960, 0xa}, 0xd3b6d4f2990, 0x0)
[2026/05/07 16:09:59 EDT] incusd: 	/build/incus/internal/server/instance/drivers/driver_common.go:1254 +0x285
[2026/05/07 16:09:59 EDT] incusd: github.com/lxc/incus/v7/internal/server/instance/drivers.(*lxc).startCommon(0xd3b6dce8300)
[2026/05/07 16:09:59 EDT] incusd: 	/build/incus/internal/server/instance/drivers/driver_lxc.go:2121 +0x1e5f
[2026/05/07 16:09:59 EDT] incusd: github.com/lxc/incus/v7/internal/server/instance/drivers.(*lxc).Start(0xd3b6dce8300, 0x0)
[2026/05/07 16:09:59 EDT] incusd: 	/build/incus/internal/server/instance/drivers/driver_lxc.go:2865 +0xd0c
[2026/05/07 16:09:59 EDT] incusd: main.instanceStart(0xd3b6da00200, {0x2a3bdd0, 0xd3b6dce8300})
[2026/05/07 16:09:59 EDT] incusd: 	/build/incus/cmd/incusd/instances.go:277 +0x382
[2026/05/07 16:09:59 EDT] incusd: main.instancesStart.func1()
[2026/05/07 16:09:59 EDT] incusd: 	/build/incus/cmd/incusd/instances.go:346 +0x1f
[2026/05/07 16:09:59 EDT] incusd: golang.org/x/sync/errgroup.(*Group).Go.func1()
[2026/05/07 16:09:59 EDT] incusd: 	/root/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50
[2026/05/07 16:09:59 EDT] incusd: created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 349
[2026/05/07 16:09:59 EDT] incusd: 	/root/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95
[2026/05/07 16:09:59 EDT] systemd: incus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
[2026/05/07 16:09:59 EDT] systemd: incus.service: Failed with result 'exit-code'.
[2026/05/07 16:09:59 EDT] systemd: incus.service: Consumed 47.949s CPU time, 261.7M memory peak.
[2026/05/07 16:09:59 EDT] kernel: audit: type=1131 audit(1778184599.996:599): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=incus comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[2026/05/07 16:10:00 EDT] systemd: incus.service: Scheduled restart job, restart counter is at 1.
[2026/05/07 16:10:00 EDT] systemd: Starting incus.service - Incus - Daemon...
[2026/05/07 16:10:00 EDT] (incusd): incus.service: Referenced but unset environment variable evaluates to an empty string: INCUS_OPTS
[2026/05/07 16:10:00 EDT] kernel: device-mapper: verity: 259:3: data block 142947 is corrupted
[2026/05/07 16:10:00 EDT] kernel: audit: type=1339 audit(1778184600.187:600): module=verity op=verify-data dev=259:3 sector=142947 res=0
[2026/05/07 16:10:00 EDT] incusd: time="2026-05-07T20:10:00Z" level=error msg="Failed getting version during QEMU initialization: exit status 127"
[2026/05/07 16:10:00 EDT] incusd: time="2026-05-07T20:10:00Z" level=warning msg="Instance type not operational" driver=qemu err="Failed getting QEMU version" type=virtual-machine
[2026/05/07 16:10:00 EDT] incusd: time="2026-05-07T20:10:00Z" level=warning msg=" - Instance type not operational, Failed getting QEMU version"
[2026/05/07 16:10:00 EDT] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-2 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules

I can search for something specific if needed

10

Depending where that message is sent, I might not have seen it.
I just did like usual,

incus admin os system update check
incus admin os system reboot

Didn’t look at console or whatsover. The server rebooted twice before coming online.

Version is 202605061755

11

QEMU related messages are probably irrelevant since no VMs are configured on this server

12

Hallo again,
Not quite sure what is happening. I changed the SSD and reinstall the OS (previous version). It installed just fine and proceed with upgrade. Now, it is looping with the following error

ERROR Failed to run: systemd-sysext refresh: exit status 1 (Failed to read metadata for image incus: Structure needs cleaning)

No config no nothing.

Does it matter that this server runs swtpm ?

14

I’m puzzled by this – based on the errors I’d say this is a hardware issue, but changing the drive would eliminate that.

I wonder if your root ext4 partition is becoming dirty somehow, which could then impact systemd-sysext from loading Incus properly. If you’re able, can you follow the instructions at Emergency Procedure for a Lost Client Certificate - IncusOS documentation to decrypt the root partition, then try running fsck and seeing if it reports fixing anything?

We do have systemd-fsck-root.service in the IncusOS image, but it doesn’t appear to be active because of a condition. It would probably be good to actually get this running, so we can be more resilient to file system errors.