Unexpected container startup delays on a single cluster node

Incus
1

I’ve started having an odd problem. I’ll start with a general description in case this sounds like a familiar problem or issue to someone.

I have two Incus clusters. The cluster in question has five cluster nodes with identical hardware.and OS. The cluster has been operating without issue, but recently the containers on one node take a long time before they can boot. Specifically, when the cluster node first boots, the “incus list” output for that node shows its container states as ERROR. After an additional 2.5 minutes or so, the problem seems to clear itself and its containers start and operate normally. Typically, each cluster node boots and has all of its containers up within about 60s. This one take 3.5m before the containers start, even though the Incus host boots normally and in its typical time. There are no disk or filesystem errors, and there is abundant filesystem space and memory. I’ve pared this host down to two containers, and still the problem persists. Containers are running on ZFS mirrored pools on solid state disks with routine scrubs and trims being done. Same for all incus hosts.

All cluster nodes are running Ubuntu 24.04.4 fully patched. Incus version is 6.0.6.

I can provide additional detail, but I’m starting simple. The OS of the containers does not seem to matter. Everything is x64.

Thanks!

2

You’ll want to look in the logs, this sounds a lot like what you’d get if a storage pool or network was determined to be unavailable at startup time. Incus will then re-check and re-try every minute, explaining this kind of delay.

3

Thanks. That gives me something pretty specific to look for.