Failing to run CNCF Distribution docker image

candlerb · August 26, 2026, 7:12am

I’m running incus 7.0.1 under Ubuntu 26.04 on a physical server.

OCI works for the “hello-world” image:

nsrc@host1:~$ incus remote list -fcsv | grep oci
oci-docker,https://docker.io,oci,none,YES,NO,NO

nsrc@host1:~$ incus launch oci-docker:hello-world --ephemeral --console
Launching the instance
Instance name is: novel-pipefish


Hello from Docker!
... etc

However, trying to run the CNCF Distribution registry it fails. I had this working last year with incus 6.0.

nsrc@host1:~$ incus launch oci-docker:registry:3 registry \
  -c boot.autorestart=true \
  -c environment.REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io --console
Launching registry
To detach from the console, press: <ctrl>+a q
DEBU[0000] using "text" logging formatter
Error: Failed running forkconsole: "container is not running: \"registry\""

nsrc@host1:~$ incus info registry --show-log
Name: registry
Description:
Status: STOPPED
Type: container (application)
Architecture: x86_64
Created: 2026/08/26 06:57 UTC
Last Used: 2026/08/26 06:57 UTC

Log (lxc.log):

lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry)
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry-1) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry-1)
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry-2) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry-2)
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry-3) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry-3)
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry-4) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry-4)
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry-5) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry-5)
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry-6) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry-6)
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry-7) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry-7)
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry-8) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry-8)
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry-9) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry-9)
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry-10) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry-10)
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry-11) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry-11)
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry-12) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry-12)
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry-13) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry-13)
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry-14) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry-14)
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry-15) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry-15)
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:507 - File exists - Creating the final cgroup 10(lxc.monitor.registry-16) failed
lxc registry 20260826065749.343 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - File exists - Failed to create monitor cgroup 10(lxc.monitor.registry-16)
lxc registry 20260826065749.386 ERROR    utils - ../src/lxc/utils.c:safe_mount:1332 - Invalid argument - Failed to mount "none" onto "/opt/incus/lib/lxc/rootfs/run"
lxc registry 20260826065749.600 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_destroy:665 - Device or resource busy - Failed to destroy 10(lxc.monitor.registry-17)
nsrc@host1:~$

Looking inside the container, I see it’s getting “refused” when it tries to do a DNS query to itself on [::]:53:

nsrc@host1:~$ incus console --show-log  registry
DEBU[0000] using "text" logging formatter
INFO[0000] redis not configured                          environment=development go.version=go1.25.9 instance.id=01a03cdc-5ba0-7366-ae31-0edb87706085 service=registry version=3.1.1
INFO[0000] using inmemory blob descriptor cache          environment=development go.version=go1.25.9 instance.id=01a03cdc-5ba0-7366-ae31-0edb87706085 service=registry version=3.1.1
INFO[0000] Starting upload purge in 0s                   environment=development go.version=go1.25.9 instance.id=01a03cdc-5ba0-7366-ae31-0edb87706085 service=registry version=3.1.1
INFO[0000] Starting cached object TTL expiration scheduler...  environment=development go.version=go1.25.9 instance.id=01a03cdc-5ba0-7366-ae31-0edb87706085 service=registry version=3.1.1
INFO[0000] PurgeUploads starting: olderThan=2026-08-19 06:57:49.473655485 +0000 UTC m=-604799.980557073, actuallyDelete=true
INFO[0000] Purge uploads finished.  Num deleted=0, num errors=1
INFO[0000] Starting upload purge in 24h0m0s              environment=development go.version=go1.25.9 instance.id=01a03cdc-5ba0-7366-ae31-0edb87706085 service=registry version=3.1.1
panic: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:39308->[::1]:53: read: connection refused

goroutine 1 [running]:
github.com/distribution/distribution/v3/registry/handlers.NewApp({0x2494900, 0xc0006947e0}, 0xc000137908)
	github.com/distribution/distribution/v3/registry/handlers/app.go:357 +0x1d45
github.com/distribution/distribution/v3/registry.NewRegistry({0x2494900?, 0xc0004699e0?}, 0xc000137908)
	github.com/distribution/distribution/v3/registry/registry.go:150 +0x56
github.com/distribution/distribution/v3/registry.init.func1(0x36e8aa0, {0xc0006b7730, 0x1, 0x1})
	github.com/distribution/distribution/v3/registry/registry.go:119 +0x106
github.com/spf13/cobra.(*Command).execute(0x36e8aa0, {0xc0006b76f0, 0x1, 0x1})
	github.com/spf13/cobra@v1.10.2/command.go:1019 +0xae7
github.com/spf13/cobra.(*Command).ExecuteC(0x36e8d80)
	github.com/spf13/cobra@v1.10.2/command.go:1148 +0x465
github.com/spf13/cobra.(*Command).Execute(0xc000002380?)
	github.com/spf13/cobra@v1.10.2/command.go:1071 +0x13
main.main()
	github.com/distribution/distribution/v3/cmd/registry/main.go:26 +0x1a

I get the same with oci-docker:distribution/distribution

I wonder if anyone has seen the same? Is this simply a bug in the container itself? But it was published 4 months ago, and I would have thought such an obvious bug would be noticed.

candlerb · August 26, 2026, 7:26am

If I run the container inside docker engine inside an incus container, it runs:

incus profile create docker
incus profile edit docker <<EOS
config:
  security.nesting: true
  security.syscalls.intercept.mknod: true
  security.syscalls.intercept.setxattr: true
description: Settings for running docker engine inside an incus container
EOS

incus launch images:ubuntu/26.04/cloud docker -p default -p docker
incus exec docker -- apt-get install docker.io
incus exec docker -- docker run -d -p 5000:5000 --name registry \
  -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
  registry:3

And it works:

incus config device add docker http proxy \
  listen=tcp:100.125.0.1:5000 connect=tcp:[::]:5000
curl http://100.125.0.1:5000/v2/
# ==> {}

So I have a workaround - but this gives you a reproducible case of a well-known OCI image which doesn’t run properly in incus.

stgraber · August 26, 2026, 1:19pm

Any chance you can try with Incus 7.3, maybe in a VM or something?
We have done some tweaks to the network setup of OCI which may fix this.

candlerb · August 26, 2026, 2:31pm

Done - it’s happy with 7.3 in a VM.