Docker no longer working inside incus container (overlayfs issue?)

Incus
1

Hmm, docker-ce is no longer working inside a freshly-created incus container:

root@registry:~# docker run -it --rm hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
4f55086f7dd0: Pull complete
d5e71e642bf5: Download complete
Digest: sha256:f9078146db2e05e794366b1bfe584a14ea6317f44027d10ef7dad65279026885
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: failed to mount /tmp/containerd-mount2123067234: mount source: "overlay", target: "/tmp/containerd-mount2123067234", fstype: overlay, flags: 0, data: "workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/5/work,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/5/fs,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/4/fs,userxattr,index=off", err: invalid argument

Run 'docker run --help' for more information

This certainly used to work - I’m using exactly the same script to create the outer container. The outer host is running incus 1:6.23-ubuntu22.04-202604261519 under Ubuntu 22.04 with kernel 5.15.0-177-generic

#!/bin/bash -eu
set -o pipefail

NODE="registry"

incus launch images:ubuntu/24.04/cloud "$NODE" -p proxmox-ct <<ECONFIG
config:
  security.nesting: true
  security.syscalls.intercept.mknod: true
  security.syscalls.intercept.setxattr: true
  cloud-init.network-config: |
    version: 2
    ethernets:
      eth0:
        dhcp4: false
        accept-ra: false
        addresses:
          - 100.64.0.90/22
        routes:
          - to: default
            via: 100.64.0.1
        nameservers:
          search: [ws.nsrc.org]
          addresses: [100.64.0.1, 8.8.8.8]
  cloud-init.user-data: |
    #cloud-config
    fqdn: ${NODE}.ws.nsrc.org
    chpasswd: { expire: False }
    users: []
    disable_root: false
    packages:
    # Required by docker-ce
    - ca-certificates
    - curl
    # Generally useful packages
    - arping
    - dnsutils
    - ftp
    - jq
    - man-db
    - manpages
    - mtr-tiny
    - nano
    - nmap
    - openssh-server
    - strace
    - sysstat
    - tcpdump
    - telnet
    - traceroute
    - unzip
    - util-linux
    - vim
    - wget
    - zip
    package_update: true
    runcmd:
    # https://docs.docker.com/engine/install/ubuntu/
    - for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do apt-get remove \$pkg || true; done
    - curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    - |
      echo "deb [arch=\$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \$(. /etc/os-release && echo "\${UBUNTU_CODENAME:-\$VERSION_CODENAME}") stable" >/etc/apt/sources.list.d/docker.list
    - apt-get update
    - apt-get -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
    - docker volume create registry-vol
    - docker run -d -p 5000:5000 -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io -v registry-vol:/var/lib/docker --restart=always --name registry registry:3
    )
ECONFIG

# Wait for cloud-init to finish
incus exec "$NODE" -- cloud-init status --wait --long
incus exec "$NODE" -- cloud-init status --format json

echo "Testing the registry..."
echo -n ":-"
for i in $(seq 1 10); do
  if curl http://registry.ws.nsrc.org:5000/v2/; then  # should respond {}
    echo
    exit 1
  fi
  sleep 1
done
echo
echo "Not running :-("
exit 1

The profile has nothing special in it:

$ incus profile show proxmox-ct
config: {}
description: Proxmox container nodes
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: wifi0
    type: nic
  kvm:
    source: /dev/kvm
    type: unix-char
  root:
    path: /
    pool: default
    type: disk
  shared:
    path: /shared
    shift: "true"
    source: /data/proxmox
    type: disk
name: proxmox-ct

I do realise that newer versions of incus can run OCI containers directly. But as I say, this used to work. I don’t know whether it’s something in incus 6.0 LTS which has changed, or something in docker-ce. (Unfortunately, I blew away the old container which was working)

But as this is a reproducible issue, I thought it was worth reporting.

EDIT: looks to be the same as #2757 (ubuntu 22.04, 5.15.0 kernel, incus 6.19.1) which the OP resolved there by moving to a newer kernel. I’ll retest on a newer system when I get a chance. It’s just odd that it used to work.