Syslog "verity: data/metadata block is corrupted"

IncusOS
1

A couple of weeks ago I moved two of my systems (backup and test) to IncusOS (I have to admit I think I really like the concept!).

About last week WebUI on backup was suddenly failing. API was still responsive. A couple of days/reboots later the system failed to boot with some “can’t decrypt error message” (flashed very briefly on screen).

I didn’t pay to much attention. Backup m.2 is an older system, the m.2 was old and well used, and this seems to have coincided with some errors I blamed some potential real I/O issues on:

2025-11-27 17:34:15.688 audit: type=1339 audit(1764261255.356:1307): module=verity op=verify-data dev=259:12 sector=137529 res=0
2025-11-27 17:34:15.688 Buffer I/O error on dev dm-3, logical block 137529, async page read
2025-11-27 17:34:15.688 audit: type=1339 audit(1764261255.356:1306): module=verity op=max-corrupted-errors dev=252:3 sector=? res=0
2025-11-27 17:34:15.688 device-mapper: verity: 259:12: reached maximum errors
2025-11-27 17:34:15.688 device-mapper: verity: 259:12: data block 137529 is corrupted
2025-11-27 17:34:15.688 audit: type=1339 audit(1764261255.355:1305): module=verity op=verify-data dev=259:12 sector=137529 res=0
2025-11-27 17:34:15.688 Buffer I/O error on dev dm-3, logical block 137529, async page read
2025-11-27 17:34:15.688 device-mapper: verity: 259:12: data block 137529 is corrupted"

[2025/11/27 17:48:17 CET] kernel: audit: type=1339 audit(1764262097.121:1314): module=verity op=verify-data dev=259:12 sector=123949 res=0
[2025/11/27 17:48:17 CET] kernel: Buffer I/O error on dev dm-3, logical block 123949, async page read
[2025/11/27 17:48:17 CET] kernel: audit: type=1339 audit(1764262097.121:1315): module=verity op=verify-data dev=259:12 sector=123949 res=0
[2025/11/27 17:48:17 CET] kernel: Buffer I/O error on dev dm-3, logical block 123949, async page read
[2025/11/27 17:48:17 CET] kernel: audit: type=1339 audit(1764262097.122:1316): module=verity op=verify-data dev=259:12 sector=123949 res=0
[2025/11/27 17:48:17 CET] kernel: Buffer I/O error on dev dm-3, logical block 123949, async page read
[2025/11/27 17:48:17 CET] kernel: audit: type=1339 audit(1764262097.122:1317): module=verity op=verify-data dev=259:12 sector=123949 res=0

I ended up replacing the m.2 with a brand new one, and reinstalling IncusOS.

Now again couple of days later I realized I have similar log messages, but on both of my systems (test and backup), probably coinciding around an update from yesterday night.

$ incus admin os debug log bak: | grep -E ‘I/O error|corrupted’

[2025/12/06 22:37:46 CET] kernel: device-mapper: verity: 259:14: data block 47556 is corrupted
[2025/12/06 22:37:46 CET] kernel: device-mapper: verity: 259:14: data block 47557 is corrupted
[2025/12/06 22:37:46 CET] kernel: device-mapper: verity: 259:14: data block 47557 is corrupted
[2025/12/06 22:37:46 CET] kernel: device-mapper: verity: 259:14: data block 38803 is corrupted
[2025/12/06 22:37:46 CET] kernel: device-mapper: verity: 259:14: data block 38803 is corrupted
[2025/12/06 22:37:46 CET] kernel: device-mapper: verity: 259:14: data block 38803 is corrupted
[2025/12/06 22:37:46 CET] kernel: device-mapper: verity: 259:14: metadata block 12 is corrupted
[2025/12/06 22:37:46 CET] kernel: device-mapper: verity: 259:14: metadata block 12 is corrupted
[2025/12/06 22:37:46 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:37:46 CET] kernel: device-mapper: verity: 259:14: metadata block 12 is corrupted
[2025/12/06 22:37:46 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:37:46 CET] kernel: device-mapper: verity: 259:14: data block 108960 is corrupted
[2025/12/06 22:37:46 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:37:47 CET] kernel: Buffer I/O error on dev dm-3, logical block 132472, async page read
[2025/12/06 22:37:47 CET] kernel: Buffer I/O error on dev dm-3, logical block 132472, async page read
[2025/12/06 22:37:47 CET] kernel: Buffer I/O error on dev dm-3, logical block 132472, async page read
[2025/12/06 22:37:47 CET] kernel: Buffer I/O error on dev dm-3, logical block 132472, async page read
[2025/12/06 22:37:47 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:37:47 CET] kernel: Buffer I/O error on dev dm-3, logical block 132472, async page read
[2025/12/06 22:37:47 CET] kernel: Buffer I/O error on dev dm-3, logical block 132472, async page read

$ incus admin os debug log test: | grep -E ‘I/O error|corrupted’

[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: data block 47542 is corrupted
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: data block 47557 is corrupted
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: data block 47557 is corrupted
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: data block 47557 is corrupted
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: metadata block 1306 is corrupted
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: metadata block 1306 is corrupted
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: data block 38803 is corrupted
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: data block 38803 is corrupted
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: data block 38803 is corrupted
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: metadata block 1306 is corrupted
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read

Or with more context on test:

[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: data block 47542 is corrupted
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: data block 47557 is corrupted
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: data block 47557 is corrupted
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: data block 47557 is corrupted
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device nvme0n1 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: metadata block 1306 is corrupted
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: metadata block 1306 is corrupted
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] incusd: SIGBUS: bus error
[2025/12/06 22:06:24 CET] incusd: PC=0x47840f m=11 sigcode=2 addr=0x26abe64
[2025/12/06 22:06:24 CET] incusd: goroutine 0 gp=0xc0007821c0 m=11 mp=0xc000780008 [idle]:
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: data block 38803 is corrupted
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: data block 38803 is corrupted
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: data block 38803 is corrupted
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] kernel: device-mapper: verity: 259:18: metadata block 1306 is corrupted
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] kernel: Buffer I/O error on dev dm-3, logical block 165552, async page read
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/06 22:06:24 CET] systemd: incus.service: Main process exited, code=killed, status=7/BUS
[2025/12/06 22:06:24 CET] systemd: incus.service: Failed with result 'signal'.
[2025/12/06 22:06:24 CET] systemd: incus.service: Unit process 2448 (dnsmasq) remains running after unit stopped.
[2025/12/06 22:06:24 CET] systemd: incus.service: Unit process 2468 (dnsmasq) remains running after unit stopped.
[2025/12/06 22:06:24 CET] systemd: incus.service: Consumed 4h 57min 48.834s CPU time, 49.8G memory peak, 184.5M memory swap peak.

Both backup and test are 2 quite different systems. Test is a 13th gen Intel with plenty of memory on an Asus board - almost exclusively running basic containers. Backup is a light 7th or 8th gen Intel U with limited memory on an Gigabyte board, running some real workloads incl. VMs with pass through.

Any thoughts on this? Are these errors something that is to be expected - on a plain linux I’d be concerned about this? Is it worth digging further, or should I just ignore this?

My goal here would be to ensure these 2 systems are happy and stable before I migrate my primary to IncusOS as well…

THANKS!!

2

@gibmat any idea of what may be going on?

It’s of course possible that there’s an actual hardware issue (bad memory, random bitflip, …) which would cause a verity failure, but the timing is a bit suspicious here.

3

It definitely looks like some sort of hardware issue happening here. My first guess would be a “dirty” mains power source or something else causing interference to the systems.

I’d start by running some diagnostics on the systems (memtest, other cpu/memory/disk benchmarks, disk-specific tests, etc) and seeing if anything turns up. It might take multiple passes too; I’ve personally seen bad RAM sticks that took a few rounds of memtest to properly identify.

I can’t think of anything IncusOS-specific right off hand. A kernel bug or similar hopefully wouldn’t be triggering on both systems, since they are sufficiently different hardware.

4

Thanks!

I’ve started MemTester86 on my test system (I have full KVM there). I don’t have direct hardware access to my backup system (without changes) - I’m running stress-ng in a container for the moment.

If this doesn’t turn up any issues I’ll go for a destructive disk test on my test system.

I’ll let you know how it goes - if you have more specific suggestions please let me know.

5

I’ve been running some heavy CPU/Memory load testing on both systems since yesterday (memtest86, stress-ng, …) => no faults yet
I have not yet performed (destructive) stress test on the NVMe…

Meanwhile I’ve been digging through some of the log files (I got similar error with the update today).

It appears the verity errors I’m seeing are from application image (/opt).
Below looks like app update is switching from 259:12/dm-4 to 159:14/dm-5 (loop4/5).
The vanity erros seem to be from OLD app image, and only AFTER the switch over.

Is it possible that the old image is being torn down/no longer available, and some stale handlers still exist, attempting to read from it (couldn’t find this in code path…)?
It seems incus.service is not stopping clean, there are at least 4 processes leftover that could be doing this access (447791, 447811, 515191, 596031)?

( I seem to be having a similar pattern for the other faults I’m seeing )

[2025/12/08 10:45:03 CET] incus-osd: 2025-12-08 10:45:03 INFO Downloading OS update version=202512080339
[2025/12/08 10:45:22 CET] incus-osd: 2025-12-08 10:45:22 INFO Applying OS update version=202512080339
[2025/12/08 10:45:22 CET] systemd: Mounting boot.mount - EFI System Partition Automount...
[2025/12/08 10:45:22 CET] kernel: kauditd_printk_skb: 2 callbacks suppressed
[2025/12/08 10:45:22 CET] kernel: audit: type=1334 audit(1765187122.886:2986): prog-id=155 op=LOAD
[...]
[2025/12/08 10:45:22 CET] kernel: audit: type=1334 audit(1765187122.889:2995): prog-id=160 op=LOAD
[2025/12/08 10:45:22 CET] systemd: Mounted boot.mount - EFI System Partition Automount.
[2025/12/08 10:45:23 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device nvme0n1 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[...]
[2025/12/08 10:45:25 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device nvme0n1 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:45:27 CET] incus-osd: 2025-12-08 10:45:27 INFO Reloading application name=incus version=202512080339
[2025/12/08 10:45:27 CET] systemd: Reload requested from client PID 764811 ('systemctl') (unit incus-osd.service)...
[2025/12/08 10:45:27 CET] systemd: Reloading...
[2025/12/08 10:45:27 CET] systemd-fstab-generator: Using verity usr device /dev/mapper/usr.
[2025/12/08 10:45:27 CET] systemd-veritysetup-generator: Using data device /dev/disk/by-partuuid/7d7a1118-d3d1-61ff-8606-35eeb6bc29f4 and hash device /dev/disk/by-partuuid/1f2dfbf3-c828-e70c-2825-eaff40e467cb for usr.
[2025/12/08 10:45:27 CET] systemd: Reloading finished in 571 ms.
[2025/12/08 10:45:27 CET] systemd: Stopping incus.service - Incus - Daemon...
[2025/12/08 10:47:27 CET] systemd: Unmounting boot.mount - EFI System Partition Automount...
[2025/12/08 10:47:27 CET] kernel: kauditd_printk_skb: 14 callbacks suppressed
[2025/12/08 10:47:27 CET] kernel: audit: type=1334 audit(1765187247.860:3010): prog-id=167 op=LOAD
[...]
[2025/12/08 10:47:27 CET] kernel: audit: type=1334 audit(1765187247.876:3019): prog-id=157 op=UNLOAD
[2025/12/08 10:47:27 CET] systemd: boot.mount: Deactivated successfully.
[2025/12/08 10:47:27 CET] systemd: Unmounted boot.mount - EFI System Partition Automount.
[2025/12/08 10:50:27 CET] systemd: var-lib-incus-storage\x2dpools-local-custom-default_backups.mount: Deactivated successfully.
[2025/12/08 10:50:27 CET] systemd: var-lib-incus-storage\x2dpools-local-custom-default_images.mount: Deactivated successfully.
[2025/12/08 10:50:28 CET] systemd: incus.service: Deactivated successfully.
[2025/12/08 10:50:28 CET] systemd: incus.service: Unit process 447791 (dnsmasq) remains running after unit stopped.
[2025/12/08 10:50:28 CET] systemd: incus.service: Unit process 447811 (dnsmasq) remains running after unit stopped.
[2025/12/08 10:50:28 CET] systemd: incus.service: Unit process 515191 (incusd) remains running after unit stopped.
[2025/12/08 10:50:28 CET] systemd: incus.service: Unit process 596031 (incusd) remains running after unit stopped.
[2025/12/08 10:50:28 CET] systemd: Stopped incus.service - Incus - Daemon.
[2025/12/08 10:50:28 CET] systemd: incus.service: Consumed 13h 15min 48.297s CPU time, 12.9G memory peak, 201.2M memory swap peak.
[2025/12/08 10:50:28 CET] kernel: kauditd_printk_skb: 14 callbacks suppressed
[2025/12/08 10:50:28 CET] kernel: audit: type=1131 audit(1765187428.021:3034): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=incus comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2025/12/08 10:50:28 CET] systemd: incus.service: Found left-over process 447791 (dnsmasq) in control group while starting unit. Ignoring.
[2025/12/08 10:50:28 CET] systemd: incus.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
[2025/12/08 10:50:28 CET] systemd: incus.service: Found left-over process 447811 (dnsmasq) in control group while starting unit. Ignoring.
[2025/12/08 10:50:28 CET] systemd: incus.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
[2025/12/08 10:50:28 CET] systemd: incus.service: Found left-over process 515191 (incusd) in control group while starting unit. Ignoring.
[2025/12/08 10:50:28 CET] systemd: incus.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
[2025/12/08 10:50:28 CET] systemd: incus.service: Found left-over process 596031 (incusd) in control group while starting unit. Ignoring.
[2025/12/08 10:50:28 CET] systemd: incus.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
[2025/12/08 10:50:28 CET] systemd: Starting incus.service - Incus - Daemon...
[2025/12/08 10:50:28 CET] (incusd): incus.service: Referenced but unset environment variable evaluates to an empty string: INCUS_OPTS
[2025/12/08 10:50:29 CET] kernel: audit: type=1325 audit(1765187429.437:3035): table=incus_test_066eb582-12b1-403b-b77e-52c9f5a8222f:60 family=2 entries=1 op=nft_register_table pid=765862 subj=unconfined comm="nft"
[2025/12/08 10:50:29 CET] kernel: audit: type=1300 audit(1765187429.437:3035): arch=c000003e syscall=46 success=yes exit=120 a0=3 a1=7ffd6354a7c0 a2=0 a3=0 items=0 ppid=765815 pid=765862 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="nft" exe="/usr/sbin/nft" subj=unconfined key=(null)
[2025/12/08 10:50:29 CET] kernel: audit: type=1327 audit(1765187429.437:3035): proctitle=6E667400637265617465007461626C6500696E6375735F746573745F30363665623538322D313262312D343033622D623737652D353263396635613832323266
[2025/12/08 10:50:29 CET] kernel: audit: type=1325 audit(1765187429.441:3036): table=incus_test_066eb582-12b1-403b-b77e-52c9f5a8222f:61 family=2 entries=1 op=nft_unregister_table pid=765863 subj=unconfined comm="nft"
[2025/12/08 10:50:29 CET] kernel: audit: type=1300 audit(1765187429.441:3036): arch=c000003e syscall=46 success=yes exit=112 a0=3 a1=7ffd74b57400 a2=0 a3=0 items=0 ppid=765815 pid=765863 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="nft" exe="/usr/sbin/nft" subj=unconfined key=(null)
[2025/12/08 10:50:29 CET] kernel: audit: type=1327 audit(1765187429.441:3036): proctitle=6E66740064656C657465007461626C6500696E6375735F746573745F30363665623538322D313262312D343033622D623737652D353263396635613832323266
[2025/12/08 10:50:29 CET] kernel: audit: type=1339 audit(1765187429.483:3037): module=verity op=verify-metadata dev=259:14 sector=0 res=0
[2025/12/08 10:50:29 CET] kernel: verity_handle_err: 90 callbacks suppressed
[2025/12/08 10:50:29 CET] kernel: device-mapper: verity: 259:12: data block 0 is corrupted
[2025/12/08 10:50:29 CET] kernel: audit: type=1339 audit(1765187429.483:3038): module=verity op=verify-data dev=259:12 sector=0 res=0
[2025/12/08 10:50:29 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-4 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:29 CET] kernel: device-mapper: verity: 259:12: data block 1 is corrupted
[2025/12/08 10:50:29 CET] kernel: audit: type=1339 audit(1765187429.487:3039): module=verity op=verify-data dev=259:12 sector=1 res=0
[2025/12/08 10:50:29 CET] kernel: device-mapper: verity: 259:12: data block 1 is corrupted
[2025/12/08 10:50:29 CET] kernel: buffer_io_error: 96 callbacks suppressed
[2025/12/08 10:50:29 CET] kernel: Buffer I/O error on dev dm-4, logical block 1, async page read
[2025/12/08 10:50:29 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-4 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:29 CET] kernel: device-mapper: verity: 259:12: data block 1 is corrupted
[2025/12/08 10:50:29 CET] kernel: Buffer I/O error on dev dm-4, logical block 1, async page read
[2025/12/08 10:50:29 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-4 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:29 CET] kernel: device-mapper: verity: 259:12: data block 1 is corrupted
[2025/12/08 10:50:29 CET] kernel: Buffer I/O error on dev dm-4, logical block 1, async page read
[2025/12/08 10:50:29 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-4 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:29 CET] kernel: device-mapper: verity: 259:12: data block 1 is corrupted
[2025/12/08 10:50:29 CET] kernel: Buffer I/O error on dev dm-4, logical block 1, async page read
[2025/12/08 10:50:29 CET] kernel: device-mapper: verity: 259:12: data block 0 is corrupted
[2025/12/08 10:50:29 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-4 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:29 CET] kernel: device-mapper: verity: 259:12: data block 1 is corrupted
[2025/12/08 10:50:29 CET] kernel: Buffer I/O error on dev dm-4, logical block 1, async page read
[2025/12/08 10:50:29 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-4 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:29 CET] kernel: device-mapper: verity: 259:12: data block 1 is corrupted
[2025/12/08 10:50:29 CET] kernel: Buffer I/O error on dev dm-4, logical block 1, async page read
[2025/12/08 10:50:29 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-4 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:29 CET] kernel: device-mapper: verity: 259:12: data block 1 is corrupted
[2025/12/08 10:50:29 CET] kernel: Buffer I/O error on dev dm-4, logical block 1, async page read
[2025/12/08 10:50:29 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-4 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:29 CET] kernel: Buffer I/O error on dev dm-4, logical block 1, async page read
[2025/12/08 10:50:29 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-4 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:29 CET] kernel: Buffer I/O error on dev dm-4, logical block 1, async page read
[2025/12/08 10:50:29 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-4 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:29 CET] kernel: Buffer I/O error on dev dm-4, logical block 1, async page read
[2025/12/08 10:50:29 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-4 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[...]
[2025/12/08 10:50:29 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-4 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:29 CET] dnsmasq: started, version 2.91 cachesize 150
[2025/12/08 10:50:29 CET] dnsmasq: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset nftset auth DNSSEC loop-detect inotify dumpfile
[2025/12/08 10:50:29 CET] dnsmasq-dhcp: DHCP, IP range 10.1.84.2 -- 10.1.84.254, lease time 1h
[2025/12/08 10:50:29 CET] dnsmasq-dhcp: DHCP, sockets bound exclusively to interface safe0
[2025/12/08 10:50:29 CET] dnsmasq: using only locally-known addresses for incus
[2025/12/08 10:50:29 CET] dnsmasq: reading /etc/resolv.conf
[2025/12/08 10:50:29 CET] dnsmasq: using nameserver 127.0.0.53#53
[2025/12/08 10:50:29 CET] dnsmasq: using only locally-known addresses for incus
[2025/12/08 10:50:29 CET] dnsmasq: read /etc/hosts - 7 names
[2025/12/08 10:50:29 CET] dnsmasq-dhcp: read /var/lib/incus/networks/safe0/dnsmasq.hosts/backup-incus.eth--1
[2025/12/08 10:50:29 CET] dnsmasq-dhcp: read /var/lib/incus/networks/safe0/dnsmasq.hosts/backup-onedrive.eth--1
[2025/12/08 10:50:29 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-4 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[...]
[2025/12/08 10:50:30 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-4 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:30 CET] kernel: device-mapper: verity: 259:12: reached maximum errors
[2025/12/08 10:50:30 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-4 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[...]
[2025/12/08 10:50:30 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-4 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:30 CET] dnsmasq-dhcp: router advertisement on fd42:d12b:4f:d391::, old prefix for incusbr0
[2025/12/08 10:50:30 CET] dnsmasq-dhcp: DHCPv6 stateless on fd42:d12b:4f:d391::, constructed for incusbr0
[2025/12/08 10:50:30 CET] dnsmasq-dhcp: DHCPv4-derived IPv6 names on fd42:d12b:4f:d391::, constructed for incusbr0
[2025/12/08 10:50:30 CET] dnsmasq-dhcp: router advertisement on fd42:d12b:4f:d391::, constructed for incusbr0
[2025/12/08 10:50:30 CET] dnsmasq: started, version 2.91 cachesize 150
[2025/12/08 10:50:30 CET] dnsmasq: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset nftset auth DNSSEC loop-detect inotify dumpfile
[2025/12/08 10:50:30 CET] dnsmasq-dhcp: DHCP, IP range 10.41.98.2 -- 10.41.98.254, lease time 1h
[2025/12/08 10:50:30 CET] dnsmasq-dhcp: DHCPv6 stateless on incusbr0
[2025/12/08 10:50:30 CET] dnsmasq-dhcp: DHCPv4-derived IPv6 names on incusbr0
[2025/12/08 10:50:30 CET] dnsmasq-dhcp: router advertisement on incusbr0
[2025/12/08 10:50:30 CET] dnsmasq-dhcp: DHCPv6 stateless on fd42:d12b:4f:d391::, constructed for incusbr0
[2025/12/08 10:50:30 CET] dnsmasq-dhcp: DHCPv4-derived IPv6 names on fd42:d12b:4f:d391::, constructed for incusbr0
[2025/12/08 10:50:30 CET] dnsmasq-dhcp: router advertisement on fd42:d12b:4f:d391::, constructed for incusbr0
[2025/12/08 10:50:30 CET] dnsmasq-dhcp: IPv6 router advertisement enabled
[2025/12/08 10:50:30 CET] dnsmasq-dhcp: DHCP, sockets bound exclusively to interface incusbr0
[2025/12/08 10:50:30 CET] dnsmasq: using only locally-known addresses for incus
[2025/12/08 10:50:30 CET] dnsmasq: reading /etc/resolv.conf
[2025/12/08 10:50:30 CET] dnsmasq: using nameserver 127.0.0.53#53
[2025/12/08 10:50:30 CET] dnsmasq: using only locally-known addresses for incus
[2025/12/08 10:50:30 CET] dnsmasq: read /etc/hosts - 7 names
[2025/12/08 10:50:30 CET] dnsmasq-dhcp: read /var/lib/incus/networks/incusbr0/dnsmasq.hosts/dbg.eth0
[2025/12/08 10:50:30 CET] dnsmasq-dhcp: read /var/lib/incus/networks/incusbr0/dnsmasq.hosts/win11.eth0
[2025/12/08 10:50:31 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device zd0 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:32 CET] systemd: var-lib-incus-storage\x2dpools-local-virtual\x2dmachines-pfsense\x2dbak.mount: Deactivated successfully.
[2025/12/08 10:50:32 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device zd0 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:32 CET] incusd: time="2025-12-08T10:50:32+01:00" level=warning msg="Failed auto start instance attempt" attempt=1 err="Failed to start device \"igb0\": Parent device 'enp1s0f0' doesn't exist" instance=pfsense-bak maxAttempts=3 project=default
[2025/12/08 10:50:37 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device zd0 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:37 CET] systemd: var-lib-incus-storage\x2dpools-local-virtual\x2dmachines-pfsense\x2dbak.mount: Deactivated successfully.
[2025/12/08 10:50:37 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device zd0 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:37 CET] incusd: time="2025-12-08T10:50:37+01:00" level=warning msg="Failed auto start instance attempt" attempt=2 err="Failed to start device \"igb0\": Parent device 'enp1s0f0' doesn't exist" instance=pfsense-bak maxAttempts=3 project=default
[2025/12/08 10:50:43 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device zd0 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:43 CET] systemd: var-lib-incus-storage\x2dpools-local-virtual\x2dmachines-pfsense\x2dbak.mount: Deactivated successfully.
[2025/12/08 10:50:43 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device zd0 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/08 10:50:43 CET] incusd: time="2025-12-08T10:50:43+01:00" level=warning msg="Failed auto start instance attempt" attempt=3 err="Failed to start device \"igb0\": Parent device 'enp1s0f0' doesn't exist" instance=pfsense-bak maxAttempts=3 project=default
[2025/12/08 10:50:43 CET] incusd: time="2025-12-08T10:50:43+01:00" level=error msg="Failed to auto start instance" err="Failed to start device \"igb0\": Parent device 'enp1s0f0' doesn't exist" instance=pfsense-bak project=default
[2025/12/08 10:50:43 CET] systemd: Started incus.service - Incus - Daemon.
[2025/12/08 10:50:43 CET] kernel: kauditd_printk_skb: 173 callbacks suppressed
[2025/12/08 10:50:43 CET] kernel: audit: type=1130 audit(1765187443.780:3169): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=incus comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
6

What do you have running on Incus at the time of the update?

In general I’d expect the kernel to keep the old image in memory until the last reference goes away, but maybe something’s being too aggressive here and breaks that somehow.

7

@gibmat may be worth spending some time testing application updates when we have a few VMs and containers running so we have open references to binaries from the previous image version.

8

Both are running a combination of containers (mainly debian 13) and VMs.

One VM is a pretty straight forward Windows 11, nothing special.

Second VM is a debian 13 build machine, coincidentally my last builds were around IncusOS.

Third VM is a pfSense (with Suricata and couple of other add-ons), it has a gei as network pass-through.

Nothing I would consider very fancy…

9

Some of those VMs have software TPMs attached then?

10

Yes, but at the moment only one of them:

$ incus config show bak:win11 --expanded
architecture: x86_64
config:
  limits.cpu: "4"
  limits.memory: 8GiB
  security.secureboot: "false"
  volatile.apply_template: copy
  volatile.cloud-init.instance-id: deb415d4-db6f-4387-ba67-dcdf218e8b39
  volatile.eth0.hwaddr: 10:66:6a:29:4c:5a
  volatile.uuid: 2eb6b538-151a-4702-a07e-176a653166a6
  volatile.uuid.generation: 2eb6b538-151a-4702-a07e-176a653166a6
devices:
  eth0:
    name: eth0
    network: incusbr0
    type: nic
  root:
    io.bus: nvme
    path: /
    pool: local
    size: 85GiB
    type: disk
  tpm:
    type: tpm
ephemeral: false
profiles:
- default
stateful: false
description: ""

At this point in time this is a pretty “empty” windows11 install.

11

@gibmat can you do a bit of stress testing on the app update code path?
I’m thinking, start a couple of containers and VMs, including proxy devices and vTPM to maximize the amount of external stuff we keep a reference on for the sysext.

Then I guess play switching the sysext back and forth between two versions and keep an eye on the kernel log for any sign of trouble?

12

OK, I think I’ve got this figured out. The update logic was always truncating any existing files when downloading an update, and if that was a sysext image it reasonably would then report issues if any reads are attempted. :slight_smile:

13

Nice! Thank you! Let me watch this a couple of days - I’ll report what I’m seeing (whether I see the problem again or not)!

14

I am having exactly the same issues. A couple days ago the WebUI stopped working, the API was half responsive (some commands would work, others wouldn’t). Today everything stopped responding so I had to cut power.

The logs show a very similar pattern: update starts, verity errors, and then general issues accessing the filesystem (like systemd-dissect not found in $PATH), and eventually incus failing:

[2025/12/10 20:55:32 WET] incusd: unexpected fault address 0x1ba9c60
[2025/12/10 20:55:32 WET] incusd: fatal error: fault
[2025/12/10 20:55:32 WET] incusd: [signal SIGBUS: bus error code=0x2 addr=0x1ba9c60 pc=0x1ba9c60]
[2025/12/10 20:55:32 WET] incusd: goroutine 430 gp=0xc0006c4e00 m=16 mp=0xc000c1c008 [running]:
[2025/12/10 20:55:32 WET] incusd: panic during panic
[2025/12/10 20:55:32 WET] incusd: SIGBUS: bus error
[2025/12/10 20:55:32 WET] incusd: PC=0x479006 m=16 sigcode=2 addr=0x479006
[2025/12/10 20:55:32 WET] incusd: goroutine 0 gp=0xc000c28000 m=16 mp=0xc000c1c008 [idle]:
[2025/12/10 20:55:32 WET] systemd: incus.service: Main process exited, code=killed, status=7/BUS
[2025/12/10 20:55:32 WET] systemd: incus.service: Failed with result 'signal'.
[2025/12/10 20:55:32 WET] systemd: incus.service: Unit process 1501 (dnsmasq) remains running after unit stopped.
[2025/12/10 20:55:32 WET] systemd: incus.service: Unit process 1571 (qemu-system-x86) remains running after unit stopped.
[2025/12/10 20:55:32 WET] systemd: incus.service: Consumed 1d 17h 25min 77ms CPU time, 10.3G memory peak, 2G memory swap peak.

Fix verity corruption warnings on application update by gibmat · Pull Request #686 · lxc/incus-os · GitHub makes it look like it is just fixing the warnings. If so, would a full filesystem failure be expected, or just warnings?

15

I think the title of that commit is wrong, it’s fixing an actual bug which will trigger failed verity reads in some cases (depending on the size of the updated application).

1 Like
16

Reporting in… I had the same verity issues again with the update 8h ago, both on my test and bak system.

bak showed extensive verity errors with a similar pattern to previous (I can share if it helps).

I’ve done a fresh install on test yesterday morning (currently trying disaster recovery on it). At the time of the issue it had a single simple debug container running (debian 13, idle), nevertheless triggered errors:

[2025/12/10 23:44:37 CET] incus-osd: 2025-12-10 23:44:37 INFO Downloading OS update version=202512100555
[2025/12/10 23:44:56 CET] incus-osd: 2025-12-10 23:44:56 INFO Applying OS update version=202512100555
[2025/12/10 23:44:56 CET] systemd: Mounting boot.mount - EFI System Partition Automount...
[2025/12/10 23:44:56 CET] kernel: kauditd_printk_skb: 2 callbacks suppressed
[2025/12/10 23:44:56 CET] kernel: audit: type=1334 audit(1765406696.886:767): prog-id=133 op=LOAD
[2025/12/10 23:44:56 CET] kernel: audit: type=1334 audit(1765406696.886:768): prog-id=111 op=UNLOAD
[2025/12/10 23:44:56 CET] kernel: audit: type=1334 audit(1765406696.886:769): prog-id=134 op=LOAD
[2025/12/10 23:44:56 CET] kernel: audit: type=1334 audit(1765406696.886:770): prog-id=135 op=LOAD
[2025/12/10 23:44:56 CET] kernel: audit: type=1334 audit(1765406696.886:771): prog-id=112 op=UNLOAD
[2025/12/10 23:44:56 CET] kernel: audit: type=1334 audit(1765406696.886:772): prog-id=113 op=UNLOAD
[2025/12/10 23:44:56 CET] kernel: audit: type=1334 audit(1765406696.887:773): prog-id=136 op=LOAD
[2025/12/10 23:44:56 CET] kernel: audit: type=1334 audit(1765406696.887:774): prog-id=122 op=UNLOAD
[2025/12/10 23:44:56 CET] kernel: audit: type=1334 audit(1765406696.887:775): prog-id=137 op=LOAD
[2025/12/10 23:44:56 CET] kernel: audit: type=1334 audit(1765406696.887:776): prog-id=115 op=UNLOAD
[2025/12/10 23:44:56 CET] systemd: Mounted boot.mount - EFI System Partition Automount.
[2025/12/10 23:44:58 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device nvme0n1 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/10 23:44:58 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device nvme0n1 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/10 23:44:59 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device nvme0n1 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/10 23:44:59 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device nvme0n1 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/10 23:44:59 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device nvme0n1 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/10 23:44:59 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device nvme0n1 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/10 23:45:01 CET] incus-osd: 2025-12-10 23:45:01 INFO Reloading application name=incus version=202512100555
[2025/12/10 23:45:01 CET] systemd: Reload requested from client PID 19939 ('systemctl') (unit incus-osd.service)...
[2025/12/10 23:45:01 CET] systemd: Reloading...
[2025/12/10 23:45:01 CET] systemd-fstab-generator: Using verity usr device /dev/mapper/usr.
[2025/12/10 23:45:01 CET] systemd-veritysetup-generator: Using data device /dev/disk/by-partuuid/b03ec821-041b-5b25-7263-c9c42ba45ed1 and hash device /dev/disk/by-partuuid/34217812-3207-cb6b-7960-ae861c8becb3 for usr.
[2025/12/10 23:45:01 CET] systemd: Reloading finished in 165 ms.
[2025/12/10 23:45:01 CET] systemd: Stopping incus.service - Incus - Daemon...
[2025/12/10 23:45:01 CET] systemd: var-lib-incus-storage\x2dpools-local-custom-default_backups.mount: Deactivated successfully.
[2025/12/10 23:45:01 CET] systemd: var-lib-incus-storage\x2dpools-local-custom-default_images.mount: Deactivated successfully.
[2025/12/10 23:45:01 CET] systemd: incus.service: Deactivated successfully.
[2025/12/10 23:45:01 CET] systemd: incus.service: Unit process 2109 (dnsmasq) remains running after unit stopped.
[2025/12/10 23:45:01 CET] systemd: incus.service: Unit process 2852 (dnsmasq) remains running after unit stopped.
[2025/12/10 23:45:01 CET] systemd: Stopped incus.service - Incus - Daemon.
[2025/12/10 23:45:01 CET] systemd: incus.service: Consumed 21.311s CPU time, 494.5M memory peak.
[2025/12/10 23:45:01 CET] systemd: incus.service: Found left-over process 2109 (dnsmasq) in control group while starting unit. Ignoring.
[2025/12/10 23:45:01 CET] systemd: incus.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
[2025/12/10 23:45:01 CET] systemd: incus.service: Found left-over process 2852 (dnsmasq) in control group while starting unit. Ignoring.
[2025/12/10 23:45:01 CET] systemd: incus.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
[2025/12/10 23:45:01 CET] systemd: Starting incus.service - Incus - Daemon...
[2025/12/10 23:45:01 CET] (incusd): incus.service: Referenced but unset environment variable evaluates to an empty string: INCUS_OPTS
[2025/12/10 23:45:01 CET] kernel: device-mapper: verity: 259:18: data block 0 is corrupted
[2025/12/10 23:45:01 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/10 23:45:01 CET] kernel: device-mapper: verity: 259:18: data block 0 is corrupted
[2025/12/10 23:45:01 CET] kernel: kauditd_printk_skb: 46 callbacks suppressed
[2025/12/10 23:45:01 CET] kernel: audit: type=1339 audit(1765406701.930:819): module=verity op=verify-data dev=259:18 sector=0 res=0
[2025/12/10 23:45:01 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/10 23:45:02 CET] kernel: device-mapper: verity: 259:18: data block 0 is corrupted
[2025/12/10 23:45:02 CET] kernel: audit: type=1339 audit(1765406701.999:820): module=verity op=verify-data dev=259:18 sector=0 res=0
[2025/12/10 23:45:02 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/10 23:45:02 CET] kernel: device-mapper: verity: 259:18: data block 0 is corrupted
[2025/12/10 23:45:02 CET] kernel: audit: type=1339 audit(1765406702.052:821): module=verity op=verify-data dev=259:18 sector=0 res=0
[2025/12/10 23:45:02 CET] 55-scsi-sg3_id.rules: WARNING: SCSI device dm-3 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2025/12/10 23:45:02 CET] kernel: audit: type=1325 audit(1765406702.220:822): table=incus:44 family=1 entries=5 op=nft_unregister_chain pid=20058 subj=unconfined comm="nft"
[2025/12/10 23:45:02 CET] kernel: audit: type=1300 audit(1765406702.220:822): arch=c000003e syscall=46 success=yes exit=144 a0=3 a1=7ffea8ccb560 a2=0 a3=0 items=0 ppid=19973 pid=20058 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="nft" exe="/usr/sbin/nft" subj=unconfined key=(null)
[2025/12/10 23:45:02 CET] kernel: audit: type=1327 audit(1765406702.220:822): proctitle=6E667400666C75736800636861696E00696E657400696E637573006677642E696E637573627230003B0064656C65746500636861696E00696E657400696E637573006677642E696E637573627230
[2025/12/10 23:45:02 CET] kernel: audit: type=1325 audit(1765406702.243:823): table=incus:45 family=1 entries=3 op=nft_unregister_chain pid=20059 subj=unconfined comm="nft"
[2025/12/10 23:45:02 CET] kernel: audit: type=1300 audit(1765406702.243:823): arch=c000003e syscall=46 success=yes exit=144 a0=3 a1=7ffeb7c5daf0 a2=0 a3=0 items=0 ppid=19973 pid=20059 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="nft" exe="/usr/sbin/nft" subj=unconfined key=(null)
[2025/12/10 23:45:02 CET] kernel: audit: type=1327 audit(1765406702.243:823): proctitle=6E667400666C75736800636861696E00696E657400696E6375730070737472742E696E637573627230003B0064656C65746500636861696E00696E657400696E6375730070737472742E696E637573627230

At this point I assume the patch didn’t make it downstream yet - or was not active in the update logic yet (I saw the patch was merged 16h ago).

17

Your IncusOS was still not on the newer version which has the fix:

[2025/12/10 23:44:37 CET] incus-osd: 2025-12-10 23:44:37 INFO Downloading OS update version=202512100555

It was released with 202512100555. On mine, when it booted after the failure it was able download that update during startup and apply it. Hopefully the next updates will go smoothly :crossed_fingers:

18

Ah, thanks! I’m on that version now (202512100555), so from here on forward I should no longer observe this issue - I’ll track and report.

19

I’ve just promoted a new stable image now which will have this fix.

20

But 202512100555 also had the fix already, right?

By the way, both IncusOS and Incus were updated to 202512112135 without any issues this time! :partying_face: