Just installed docker into unprivileged incus instance.
To my surprise even with ZFS as incus back-end, docker inside runs with overlay2 storage driver! It seems thanks to recent upgrades in ZFS 2.2.2? Have no idea… . Everything works so far, only there is a noticeable delay while creating docker containers and of course warning by docker. Googling around the issue is not much help (seems like a pretty new feature) Native Overlay Diff: false is a main suspect. I followed some guides around the internet to pass redirect_dir=off parameter to overlay kernel module but without success.
My host is Debian 12 + zfs + incus 0.4
azd@winterfell:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 12 (bookworm)
Release: 12
Codename: bookworm
azd@winterfell:~$ uname -a
Linux winterfell 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux
azd@winterfell:~$ sudo zfs version
zfs-2.2.2-3~bpo12+1
zfs-kmod-2.2.2-3~bpo12+1
azd@winterfell:~$ incus version
Client version: 0.4
Server version: 0.4
I have set overlay parameters as found in some older guides on the internet, did reboot but no success.
azd@winterfell:/etc/modprobe.d$ cat disable_overlay_redirect_dir.conf
options overlay metacopy=off redirect_dir=off
azd@winterfell:/etc/modprobe.d$
my instance for hosting docker
azd@winterfell:~$ incus config show dockerhost
architecture: x86_64
config:
image.architecture: amd64
image.description: Debian bookworm amd64 (20240120_05:24)
image.os: Debian
image.release: bookworm
image.serial: "20240120_05:24"
image.type: squashfs
image.variant: default
security.nesting: "true"
security.syscalls.intercept.mknod: "true"
security.syscalls.intercept.setxattr: "true"
volatile.base_image: 2761d218a6600f8e914017544388e761fd3cb64e6c3523e1b87f5cab75814f47
volatile.cloud-init.instance-id: cfcf7d0c-e028-4487-a2a3-93fdd7bf94ff
volatile.eth0.host_name: veth7fde89d1
volatile.eth0.hwaddr: 00:16:3e:05:ba:32
volatile.idmap.base: "0"
volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.idmap: '[]'
volatile.last_state.power: RUNNING
volatile.last_state.ready: "false"
volatile.uuid: d509899d-6222-491d-a44a-2cf63266c626
volatile.uuid.generation: d509899d-6222-491d-a44a-2cf63266c626
devices: {}
ephemeral: false
profiles:
- dmz33
stateful: false
description: ""
inside the instance when running systemsctl status docker
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.074737554Z" level=info msg="Loading containers: start."
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.075320837Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: , error: exec: \"modprobe\": executable file not found in $PATH"
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.307565319Z" level=info msg="Loading containers: done."
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.321088589Z" **level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: running in a user namespace" storage-driver=overlay2**
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.321364256Z" level=warning msg="WARNING: bridge-nf-call-iptables is disabled"
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.321386666Z" level=warning msg="WARNING: bridge-nf-call-ip6tables is disabled"
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.321416301Z" level=info msg="Docker daemon" commit=615dfdf containerd-snapshotter=false storage-driver=overlay2 version=25.0.0
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.321963440Z" level=info msg="Daemon has completed initialization"
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.402473726Z" level=info msg="API listen on /run/docker.sock"
Jan 20 16:08:30 dockerhost systemd[1]: Started docker.service - Docker Application Container Engine.
docker info output
root@dockerhost:~# docker info
Client: Docker Engine - Community
Version: 25.0.0
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.12.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.24.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 25.0.0
Storage Driver: overlay2
Backing Filesystem: zfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: false
userxattr: true
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
runc version: v1.1.11-0-g4bccb38
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.1.0-17-amd64
Operating System: Debian GNU/Linux 12 (bookworm)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 31.16GiB
Name: dockerhost
ID: 82fbd493-b7c1-4a63-a18b-3761559bda88
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Anyone with incus + docker on zfs with the same issue and possible solutions?