Image is not found in cluster

1

I am trying to spawn a container in a incus cluster, however it looks like the image is not found in one of the cluster nodes:

# incus init fedora-40-image  my-instance 
Creating my-instance
Error: Failed creating instance from image: open /var/lib/incus/images/9722928c09f8170148926377a98e792eb0da80d31bacef21042eebac5c021640: no such file or directory

If I target other cluster member then it works:

# incus init fedora-40-image  my-instance  --target host1
Creating my-instance

The node where it fails was actually added after the image was created first in the cluster (which by then had only 2 members). I would have expected that the image is distributed automatically among the cluster members (or at least on demand when needed). However this doesn’t seem to be the case.
Am I missing something?
Thanks!

2

Yeah, it would normally transfer it in the background the first time it’s needed.

It sounds like in this case that process must have failed in a weird way and now the system thinks it has the image when it in fact doesn’t.

incus admin sql global "SELECT images_nodes.id, images.fingerprint, nodes.name FROM images_nodes JOIN images ON images_nodes.image_id=images.id JOIN nodes ON images_nodes.node_id=nodes.id"

That will show you all your images and what server Incus thinks they’re available on.

In your case, look for your image hash (972…) and find the entry that says it’s on the system that doesn’t actually have it.

You can then do incus admin sql global "DELETE FROM images_nodes WHERE id=XYZ replacing XYZ with the row id you want to get rid of.

Doing that should then cause Incus to fetch the image from one of the other systems automatically the next time you try to use the image.

3

That indeed worked. Thanks!
I wonder how is it possible that incus got into an inconsistent state. My guess is that there is a bug in incus when adding a new member to the cluster, but I cannot really say how to reproduce the error.

4

I now realize that none of the images where in the new cluster member. The only thing I can think of is that there is an assumption in incus that at least three members of the cluster cache the images and all the images created when the cluster had only 2 members where automatically assumed to be in the three members when the third member was added.