Incus + Docker unpriviledged on ZFS - Native Overlay Diff: false warning

Just installed docker into unprivileged incus instance.

To my surprise even with ZFS as incus back-end, docker inside runs with overlay2 storage driver! It seems thanks to recent upgrades in ZFS 2.2.2? Have no idea… . Everything works so far, only there is a noticeable delay while creating docker containers and of course warning by docker. Googling around the issue is not much help (seems like a pretty new feature) Native Overlay Diff: false is a main suspect. I followed some guides around the internet to pass redirect_dir=off parameter to overlay kernel module but without success.

My host is Debian 12 + zfs + incus 0.4

azd@winterfell:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm

azd@winterfell:~$ uname -a
Linux winterfell 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux

azd@winterfell:~$ sudo zfs version
zfs-2.2.2-3~bpo12+1
zfs-kmod-2.2.2-3~bpo12+1

azd@winterfell:~$ incus version
Client version: 0.4
Server version: 0.4

I have set overlay parameters as found in some older guides on the internet, did reboot but no success.

azd@winterfell:/etc/modprobe.d$ cat disable_overlay_redirect_dir.conf
options overlay metacopy=off redirect_dir=off
azd@winterfell:/etc/modprobe.d$

my instance for hosting docker

azd@winterfell:~$ incus config show dockerhost
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Debian bookworm amd64 (20240120_05:24)
  image.os: Debian
  image.release: bookworm
  image.serial: "20240120_05:24"
  image.type: squashfs
  image.variant: default
  security.nesting: "true"
  security.syscalls.intercept.mknod: "true"
  security.syscalls.intercept.setxattr: "true"
  volatile.base_image: 2761d218a6600f8e914017544388e761fd3cb64e6c3523e1b87f5cab75814f47
  volatile.cloud-init.instance-id: cfcf7d0c-e028-4487-a2a3-93fdd7bf94ff
  volatile.eth0.host_name: veth7fde89d1
  volatile.eth0.hwaddr: 00:16:3e:05:ba:32
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: d509899d-6222-491d-a44a-2cf63266c626
  volatile.uuid.generation: d509899d-6222-491d-a44a-2cf63266c626
devices: {}
ephemeral: false
profiles:
- dmz33
stateful: false
description: ""

inside the instance when running systemsctl status docker

Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.074737554Z" level=info msg="Loading containers: start."
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.075320837Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: , error: exec: \"modprobe\": executable file not found in $PATH"
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.307565319Z" level=info msg="Loading containers: done."
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.321088589Z" **level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: running in a user namespace" storage-driver=overlay2**
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.321364256Z" level=warning msg="WARNING: bridge-nf-call-iptables is disabled"
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.321386666Z" level=warning msg="WARNING: bridge-nf-call-ip6tables is disabled"
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.321416301Z" level=info msg="Docker daemon" commit=615dfdf containerd-snapshotter=false storage-driver=overlay2 version=25.0.0
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.321963440Z" level=info msg="Daemon has completed initialization"
Jan 20 16:08:30 dockerhost dockerd[4582]: time="2024-01-20T16:08:30.402473726Z" level=info msg="API listen on /run/docker.sock"
Jan 20 16:08:30 dockerhost systemd[1]: Started docker.service - Docker Application Container Engine.

docker info output

root@dockerhost:~# docker info
Client: Docker Engine - Community
 Version:    25.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 25.0.0
 Storage Driver: overlay2
  Backing Filesystem: zfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc version: v1.1.11-0-g4bccb38
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-17-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 31.16GiB
 Name: dockerhost
 ID: 82fbd493-b7c1-4a63-a18b-3761559bda88
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Anyone with incus + docker on zfs with the same issue and possible solutions?

I do not clearly understand what do you ask, where do you see any errors? IMHO everything works as it should as I read logs you have posted. Why do you think you need native diff for overlay2 and what for?

it is generally needed for better performance while creating docker containers.

@zdenekv Did you find the solution to this?

Somewhere I found this command:

incus storage volume set default container/{name} zfs.delegate=true

It requires latest ZFS stable and I was able to run docker containers without errors. However, I’m not sure if my containers have made use of it.

Lastly I would suggest looking into the latest Incus release 6.3 which supports OCI natively now, see release notes. Still early but I was able to convert quite a few of my docker containers into native Incus OCI.