Rootless docker on new Ubuntu kernels does not work

Incus
,
1

I just found that rootless docker is unable to work on ZFS devices which are VFS idmap shifted.

The log says: [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: permission denied

The command it tries to run is: rootlesskit --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh

Any suggestions what may be a reason and is it possible to fix it?

I tried Ubuntu Jammy and Debian Bookworm as unprivileged containers with security nesting enabled. Host is Ubuntu with ubuntu kernel 6.6.0-14.

I made an experiment and replaced filesystem for docker rootless user with btrfs and there is no change in dockers behaviour, the same error.

2

Can you look at the output of sudo dmesg on the host for any DENIED entries?
The permission denied here looks a bit odd.

3

You are correct, I’ve should notice that, but I haven’t.

[76986.164765] audit: type=1400 audit(1705375331.807:282): apparmor="DENIED" operation="userns_create" class="namespace" info="User namespace creation restricted" error=-13 namespace="root//incus-dockerz_<var-lib-incus>" profile="unconfined" pid=2652411 comm="rootlesskit" requested="userns_create" denied="userns_create"

What is the reason that apparmour is blocking it? incus 0.4-202312232115-ubuntu22.04

I have also noticed more blockades related to other containers which seemed to work fine but it looks that they are not in fully good condition:

[64889.282396] audit: type=1400 audit(1705363235.130:281): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="incus-pentylon_</var/lib/incus>" name="/run/systemd/unit-root/proc/" pid=2218164 comm="(ogrotate)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
4

Ah, you’re hitting the new weird Ubuntu-only apparmor behavior…

This has apparently caused a fair bit of breakage: Bug #2046844 “AppArmor user namespace creation restrictions caus...” : Bugs : apparmor package : Ubuntu

Note that this is a weird change of behavior that’s only part of the Ubuntu kernel, other distributions do not do that, nor does running a non-Ubuntu kernel on Ubuntu.

The recommended solution for this is apparently to create an apparmor profile for the application (rootlesskit) which then allows userns_create. I’m not sure if this is something that can be used from within a container though.

Another option is to not use the Ubuntu kernel but use a clean upstream kernel instead.
I do make such kernel packages if you want to go this route. Given you’re using ZFS, you’d need both:

1 Like
5

Thank you very much, Stephane, for detailed explanation. Reading the ticket on launchpad I am shocked how reckless is Mr Johansen in his experiments made without any plan and without caring about consequences.

I need to investigate which of paths you have suggested I will follow, I will share my experiences soon.

6

So my decision was to replace Ubuntu with Bookworm as for today it’s newer than Jammy and has similar perspectives of long term support plus replaced standard Debian kernel with Zabbly one.

After reading Ubuntu tickets I rejected going forward with Ubuntu kernel as it’s simply unpredictable and patching it with own apparmor rules may be time consuming and surprising.

Rootless docker works now, with ZFS VFS uid shifting as I planned.

7

Great to hear!

1 Like