Docker OCI images not working with cluster system's cache: 'Failed getting remote image info: Image not found'

Incus
, ,
1

Hello, i would like to work with Docker images on my cluster system, but have been running into some issues when trying to launch basic instances. The workflow that triggers this errors is the following:

incus launch docker:alpine test --target=amd01

incus delete test --force 

incus launch docker:alpine test --target=amd01

Error: Failed instance creation: Failed getting remote image info: Image not found

The error also occurs if we replace the delete command with a stop command.

Some things that i have found while trying to debug this are that:

  1. While the instance is not yet deleted(or stopped) for the first time in a specific target, the creation can happen without errors.
  2. The error only happens if the instance is initiated(if i create instead of launch, delete and create again it works fine).
  3. If i launch with the fingerprint of a cached image instead of using docker:image, it also works fine.

Also i am currently running these commands inside a project thats not default.

Is there anything i can do to avoid this without using the direct fingerprint or giving aliases to images?

2 
stgraber@castiana:~ (incus:s-shf-cluster/stgraber-dev)$ incus launch oci-docker:alpine test --target=chulak
Launching test
stgraber@castiana:~ (incus:s-shf-cluster/stgraber-dev)$ incus delete test --force 
stgraber@castiana:~ (incus:s-shf-cluster/stgraber-dev)$ incus launch oci-docker:alpine test --target=chulak
Launching test
stgraber@castiana:~ (incus:s-shf-cluster/stgraber-dev)$ incus version
Client version: 6.22
Server version: 6.22
stgraber@castiana:~ (incus:s-shf-cluster/stgraber-dev)$

The above is running on a remote cluster (s-shf-cluster) in a non-default project (stgraber-dev).

So it would appear that there’s more to this issue than what you’ve listed above…
At first I thought it was because of image replication in the cluster, but since both launches are on the same server, that’s unlikely to be the issue.

3

Btw, @iurihol00 is a student of mine and this test was at the university’s cluster we maintain. Thanks for your test results.

I’m trying to figure out exactly what’s going on. I’m installing IncusOS on my homelab to see if I could reproduce it here, but I still have some issues to figure out before that.

It’s a weird issue that’s happening here. It is 100% reproducible, but we’re only testing it at our cluster. I would like to add some extra information on this issue as it’s kind of annoying, before I can do further testing.

The cluster is only accessed from an incus container instance named bastion which runs ubuntu jammy. The cluster itself is running on debian bookworm. I’ve had similar issues before and it reduced when I installed skopeo and umoci both on the servers and on the remote client. Right now, I checked the version information for them and got the following:

pargo@bastion:~/controller/cluster/ansible$ incus version
Client version: 6.22
Server version: 6.22
pargo@bastion:~/controller/cluster/ansible$ skopeo --version
skopeo version 1.4.1
pargo@bastion:~/controller/cluster/ansible$ ssh amd01 -- skopeo --version
skopeo version 1.9.3
pargo@bastion:~/controller/cluster/ansible$ umoci --version
umoci version 0.4.7+ds-2ubuntu0.1
pargo@bastion:~/controller/cluster/ansible$ ssh amd01 -- umoci --version
umoci version 0.4.7+ds-3+b7

My current guess is that skopeo is sometimes called from the client side and sometimes on the server side. I’ll try to test it further.

4

This should be a code path which is completely server-side.

Operations like incus image info docker:hello-world or incus image export docker:hello-world would be entirely client side and require skopeo and umoci on the client.

But incus launch docker:hello-world instead results (per --debug) in a server-side request like:

	{
		"architecture": "",
		"config": {},
		"devices": {},
		"ephemeral": false,
		"profiles": null,
		"stateful": false,
		"description": "",
		"name": "test",
		"source": {
			"type": "image",
			"certificate": "",
			"alias": "alpine",
			"server": "https://docker.io",
			"protocol": "oci",
			"mode": "pull",
			"allow_inconsistent": false
		},
		"instance_type": "",
		"type": "container",
		"start": true
	}

This then causes the server to use skopeo to fetch the files and umoci to repack them. Though in the scenario above, it should only result in the server resolving the image through skopeo then noticing it already has that image in the image store and instead using that image rather than downloading and repacking it again.

5

Additional data points that may be useful:

What does incus info | grep server_name show on the bastion?
I’m basically wondering if you’re talking directly to your target server or to another server which will then forward the request.

In my case, I was targeting chulak but my client was talking through sateda

Do you see the cached image in incus image list for that project?

stgraber@castiana:~ (incus:s-shf-cluster/stgraber-dev)$ incus info | grep server_name
  server_name: sateda
stgraber@castiana:~ (incus:s-shf-cluster/stgraber-dev)$ incus launch oci-docker:alpine test --target=chulak^C
stgraber@castiana:~ (incus:s-shf-cluster/stgraber-dev)$ incus delete test --force 
stgraber@castiana:~ (incus:s-shf-cluster/stgraber-dev)$ incus info | grep server_name
  server_name: sateda
stgraber@castiana:~ (incus:s-shf-cluster/stgraber-dev)$ incus launch oci-docker:alpine test --target=chulak
Launching test
stgraber@castiana:~ (incus:s-shf-cluster/stgraber-dev)$ incus image list | grep -i alpine
|       | 71a4daca3119 | no     | docker.io/library/alpine (OCI)       | aarch64      | CONTAINER       | 4.00MiB   | 2026/03/23 14:29 EDT |
|       | a302fc0f4dba | no     | docker.io/library/alpine (OCI)       | x86_64       | CONTAINER       | 3.68MiB   | 2026/03/23 15:09 EDT |

Do you have skopeo and umoci on all the servers?
Note that if you’re using the Zabbly packages, then the answer would be yes as they are always bundled in /opt/incus/bin/

6

I was trying to look for a clue on the cluster database, but haven’t found anything yet.

pargo@bastion:~$ incus info | grep server_name
  server_name: amd01

Skopeo and umoci are installed on all server machines:

pargo@bastion:~$ for i in {amd01,amd02,amd03,amd04,intel01,intel02,intel03,gpu01,gpu02}; do echo "$i"; ssh $i -- skopeo --version; ssh $i -- umoci --version; done
amd01
skopeo version 1.9.3
umoci version 0.4.7+ds-3+b7
amd02
skopeo version 1.9.3
umoci version 0.4.7+ds-3+b7
amd03
skopeo version 1.9.3
umoci version 0.4.7+ds-3+b7
amd04
skopeo version 1.9.3
umoci version 0.4.7+ds-3+b7
intel01
skopeo version 1.9.3
umoci version 0.4.7+ds-3+b7
intel02
skopeo version 1.9.3
umoci version 0.4.7+ds-3+b7
intel03
skopeo version 1.9.3
umoci version 0.4.7+ds-3+b7
gpu01
skopeo version 1.9.3
umoci version 0.4.7+ds-3+b7
gpu02
skopeo version 1.9.3
umoci version 0.4.7+ds-3+b7

We’re connected directly to amd01. But this issue is reproducible on any target.

pargo@bastion:~$ incus image list
+----------------------+--------------+--------+------------------------------------------------+--------------+-----------+------------+----------------------+
|        ALIAS         | FINGERPRINT  | PUBLIC |                  DESCRIPTION                   | ARCHITECTURE |   TYPE    |    SIZE    |     UPLOAD DATE      |
+----------------------+--------------+--------+------------------------------------------------+--------------+-----------+------------+----------------------+
(...)
+----------------------+--------------+--------+------------------------------------------------+--------------+-----------+------------+----------------------+
|                      | 6f1d9ab81ed6 | no     | docker.io/library/hello-world (OCI)            | x86_64       | CONTAINER | 0.00MiB    | 2026/03/23 17:27 -03 |
+----------------------+--------------+--------+------------------------------------------------+--------------+-----------+------------+----------------------+
(...)
pargo@bastion:~$ incus image delete 6f1d9ab81ed6
pargo@bastion:~$ incus launch docker:hello-world hw --ephemeral --target amd01
Launching hw
pargo@bastion:~$ incus launch docker:hello-world hw --ephemeral --target amd01
Launching hw
Error: Failed instance creation: Failed getting remote image info: Image not found
pargo@bastion:~$ incus launch docker:hello-world hw --ephemeral --target amd02
Launching hw
pargo@bastion:~$ incus launch docker:hello-world hw --ephemeral --target amd02
Launching hw
pargo@bastion:~$ incus launch docker:hello-world hw --ephemeral --target amd02
Launching hw
Error: Failed instance creation: Failed getting remote image info: Image not found
pargo@bastion:~$ incus launch docker:hello-world hw --ephemeral --target amd03
Launching hw
pargo@bastion:~$ incus launch docker:hello-world hw --ephemeral --target amd03
Launching hw
pargo@bastion:~$ incus launch docker:hello-world hw --ephemeral --target amd03
Launching hw
Error: Failed instance creation: Failed getting remote image info: Image not found
pargo@bastion:~$ incus create docker:hello-world hw --ephemeral --target amd04
Creating hw
pargo@bastion:~$ incus delete hw                   
pargo@bastion:~$ incus create docker:hello-world hw --ephemeral --target amd04 && incus delete hw
Creating hw
pargo@bastion:~$ incus create docker:hello-world hw --ephemeral --target amd04 && incus delete hw
Creating hw
pargo@bastion:~$ incus create docker:hello-world hw --ephemeral --target amd04 && incus delete hw
Creating hw
pargo@bastion:~$ incus create docker:hello-world hw --ephemeral --target amd04 && incus delete hw
Creating hw
pargo@bastion:~$ incus create docker:hello-world hw --ephemeral --target amd04 && incus delete hw
Creating hw
pargo@bastion:~$ incus create docker:hello-world hw --ephemeral --target amd04 && incus delete hw
Creating hw
pargo@bastion:~$ incus launch docker:hello-world hw --ephemeral --target amd04
Launching hw
pargo@bastion:~$ incus launch docker:hello-world hw --ephemeral --target amd04
Launching hw
Error: Failed instance creation: Failed getting remote image info: Image not found

What we can see is that the first time it is run on a server, it is downloaded either through docker.io or from another cluster member, and that command always works. The second command usually fails, even though the image is cached onto that cluster node.

I actually noticed something different from our previous experiments today. A few times, it failed only on the third attempt for some reason.

Btw, looking at the incus monitor, I found something weird when running it directly from the server.

pargo@amd01:~$ skopeo --insecure-policy inspect docker://docker.io/hello-world
FATA[0000] Error parsing image name "docker://docker.io/hello-world": pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:53022->[::1]:53: read: connection refused

It might be something related to the university’s firewall. There is indeed a skopeo command failing. Why it works sometimes and why it doesn’t work sometimes still puzzles me.

I did find something which might be local, so don’t worry too much about it. After spending the whole day on this, I’ll get back to it on another day.

7

Thinking about it now, there are a few things which seem odd. First is that I’m guessing the access is rate-limited, for some reason. But when the image is not on the node cache, it gets it from another machine without using skopeo to get its fingerprint. And it can launch just fine. But when it is in local cache, it is querying skopeo, which triggers the error.

There is also the difference of create and launch, which have different behaviours. Many creates don’t triggr the rate limit, but one launch does. Then creates stop working.

And there is also the issue that launch breaks skopeo for some reason.

I can’t explain what happened just now.

pargo@amd01:~$ skopeo --insecure-policy inspect docker://docker.io/hello-world                                                                                                                                                                                                  
{                                                                                                                                                                                                                                                                               
    "Name": "docker.io/library/hello-world",                                                                                                                                                                                                                                    
    "Digest": "sha256:6ae972d1ac530d897fd03238e11e16157498834b49574f21a0c7d0f51b58f84d",                                                                                                                                                                                        
    "RepoTags": [                                                                                                                                                                                                                                                               
        "latest",                                                                                                                                                                                                                                                               
        "linux",                                                                                                                                                                                                                                                                
        "nanoserver",                                                                                                                                                                                                                                                           
        "nanoserver-1709",                                                                                                                                                                                                                                                      
        "nanoserver-1803",                                                                                                                                                                                                                                                      
        "nanoserver-1809",                                                                                                                                                                                                                                                      
        "nanoserver-ltsc2022",                                                                                                                                                                                                                                                  
        "nanoserver-ltsc2025",                                                                                                                                                                                                                                                  
        "nanoserver-sac2016",                                                                                                                                                                                                                                                   
        "nanoserver1709"                                                                                                                                                                                                                                                        
    ],                                                                                                                                                                                                                                                                          
    "Created": "2026-03-23T21:33:59.562202219Z",                                                                                                                                                                                                                                
    "DockerVersion": "",                                                                                                                                                                                                                                                        
    "Labels": null,                                                                                                                                                                                                                                                             
    "Architecture": "amd64",                                                                                                                                                                                                                                                    
    "Os": "linux",                                                                                                                                                                                                                                                              
    "Layers": [                                                                                                                                                                                                                                                                 
        "sha256:4f55086f7dd096d48b0e49be066971a8ed996521c2e190aa21b2435a847198b4"                                                                                                                                                                                               
    ],                                                                                                                                                                                                                                                                          
    "Env": [                                                                                                                                                                                                                                                                    
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"                                                                                                                                                                                                     
    ]                                                                                                                                                                                                                                                                           
}                                                                                                                                                                                                                                                                               
pargo@amd01:~$ skopeo --insecure-policy inspect docker://docker.io/hello-world                                                                                                                                                                                        [144/1883]
{                                                                                                                                                                                                                                                                               
    "Name": "docker.io/library/hello-world",                                                                                                                                                                                                                                    
    "Digest": "sha256:6ae972d1ac530d897fd03238e11e16157498834b49574f21a0c7d0f51b58f84d",                                                                                                                                                                                        
    "RepoTags": [                                                                                                                                                                                                                                                               
        "latest",                                                                                                                                                                                                                                                               
        "linux",                                                                                                                                                                                                                                                                
        "nanoserver",                                                                                                                                                                                                                                                           
        "nanoserver-1709",                                                                                                                                                                                                                                                      
        "nanoserver-1803",                                                                                                                                                                                                                                                      
        "nanoserver-1809",                                                                                                                                                                                                                                                      
        "nanoserver-ltsc2022",                                                                                                                                                                                                                                                  
        "nanoserver-ltsc2025",                                                                                                                                                                                                                                                  
        "nanoserver-sac2016",                                                                                                                                                                                                                                                   
        "nanoserver1709"                                                                                                                                                                                                                                                        
    ],                                                                                                                                                                                                                                                                          
    "Created": "2026-03-23T21:33:59.562202219Z",
    "DockerVersion": "",
    "Labels": null,
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:4f55086f7dd096d48b0e49be066971a8ed996521c2e190aa21b2435a847198b4"
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
}
pargo@amd01:~$ skopeo --insecure-policy inspect docker://docker.io/hello-world
{
    "Name": "docker.io/library/hello-world",
    "Digest": "sha256:6ae972d1ac530d897fd03238e11e16157498834b49574f21a0c7d0f51b58f84d",
    "RepoTags": [
        "latest",
        "linux",
        "nanoserver",
        "nanoserver-1709",
        "nanoserver-1803",
        "nanoserver-1809",
        "nanoserver-ltsc2022",
        "nanoserver-ltsc2025",
        "nanoserver-sac2016",
        "nanoserver1709"
    ],
    "Created": "2026-03-23T21:33:59.562202219Z",
    "DockerVersion": "",
    "Labels": null,
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:4f55086f7dd096d48b0e49be066971a8ed996521c2e190aa21b2435a847198b4"
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
}
pargo@amd01:~$ skopeo --insecure-policy inspect docker://docker.io/hello-world
{
    "Name": "docker.io/library/hello-world",
    "Digest": "sha256:6ae972d1ac530d897fd03238e11e16157498834b49574f21a0c7d0f51b58f84d",
    "RepoTags": [
        "latest",
        "linux",
        "nanoserver",
        "nanoserver-1709",
        "nanoserver-1803",
        "nanoserver-1809",
        "nanoserver-ltsc2022",
        "nanoserver-ltsc2025",
        "nanoserver-sac2016",
        "nanoserver1709"
    ],
    "Created": "2026-03-23T21:33:59.562202219Z",
    "DockerVersion": "",
    "Labels": null,
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:4f55086f7dd096d48b0e49be066971a8ed996521c2e190aa21b2435a847198b4"
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
}
pargo@amd01:~$ skopeo --insecure-policy inspect docker://docker.io/hello-world                                                                                                                                                                                                  
{                                                                                                                                                                                                                                                                               
    "Name": "docker.io/library/hello-world",                                                                                                                                                                                                                                    
    "Digest": "sha256:6ae972d1ac530d897fd03238e11e16157498834b49574f21a0c7d0f51b58f84d",                                                                                                                                                                                        
    "RepoTags": [                                                                                                                                                                                                                                                               
        "latest",                                                                                                                                                                                                                                                               
        "linux",                                                                                                                                                                                                                                                                
        "nanoserver",                                                                                                                                                                                                                                                           
        "nanoserver-1709",                                                                                                                                                                                                                                                      
        "nanoserver-1803",                                                                                                                                                                                                                                                      
        "nanoserver-1809",
        "nanoserver-ltsc2022",
        "nanoserver-ltsc2025",
        "nanoserver-sac2016",
        "nanoserver1709"
    ],
    "Created": "2026-03-23T21:33:59.562202219Z",
    "DockerVersion": "",
    "Labels": null,
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:4f55086f7dd096d48b0e49be066971a8ed996521c2e190aa21b2435a847198b4"
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
}
pargo@amd01:~$ incus launch docker:hello-world hw --ephemeral                                                                            
A lançar hw
pargo@amd01:~$ skopeo --insecure-policy inspect docker://docker.io/hello-world
FATA[0000] Error parsing image name "docker://docker.io/hello-world": pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:53516->[::1]:53: read: connection refused 
pargo@amd01:~$ skopeo --insecure-policy inspect docker://docker.io/hello-world
FATA[0000] Error parsing image name "docker://docker.io/hello-world": pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:40506->[::1]:53: read: connection refused 
pargo@amd01:~$ skopeo --insecure-policy inspect docker://docker.io/hello-world
FATA[0000] Error parsing image name "docker://docker.io/hello-world": pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:33320->[::1]:53: read: connection refused 
pargo@amd01:~$ skopeo --insecure-policy inspect docker://docker.io/hello-world
FATA[0000] Error parsing image name "docker://docker.io/hello-world": pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:44620->[::1]:53: read: connection refused 
pargo@amd01:~$ skopeo --insecure-policy inspect docker://docker.io/hello-world
FATA[0000] Error parsing image name "docker://docker.io/hello-world": pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:49097->[::1]:53: read: connection refused 
pargo@amd01:~$ incus launch docker:hello-world hw --ephemeral                                                                            
A lançar hw
Erro: Failed instance creation: Image not found
pargo@amd01:~$ incus launch docker:hello-world hw --ephemeral
A lançar hw
Erro: Failed instance creation: Image not found
pargo@amd01:~$ incus launch docker:hello-world hw --ephemeral --target amd02
A lançar hw
pargo@amd01:~$ incus launch docker:hello-world hw --ephemeral --target amd02
A lançar hw
pargo@amd01:~$ incus launch docker:hello-world hw --ephemeral --target amd02
A lançar hw
Erro: Failed instance creation: Failed getting remote image info: Image not found
pargo@amd01:~$ incus create docker:hello-world hw --ephemeral --target amd02                                                             
Criando hw
Erro: Failed instance creation: Failed getting remote image info: Image not found
pargo@amd01:~$ incus create docker:hello-world hw --ephemeral --target amd03 && incus delete hw
Criando hw
pargo@amd01:~$ incus create docker:hello-world hw --ephemeral --target amd03 && incus delete hw
Criando hw
pargo@amd01:~$ incus create docker:hello-world hw --ephemeral --target amd03 && incus delete hw
Criando hw
pargo@amd01:~$ incus launch docker:hello-world hw --ephemeral --target amd03
A lançar hw
pargo@amd01:~$ incus create docker:hello-world hw --ephemeral --target amd03 && incus delete hw
Criando hw
Erro: Failed instance creation: Failed getting remote image info: Image not found

There seems to be different ways in which they are created, depending on the command which is run. And it affects connectivity with skopeo after the command is run from that node.

8

Could it be as simple as round robin DNS and your firewall only letting one of the IP addresses out?