`failed to write a *:* rwm to devices.allow ... operation not permitted` in privileged container

LXD
#1

Hey, I run a kubelet in a privileged LXD container. I launch the container as follows:

[...]
  read -r -d '' raw_lxc <<RAW_LXC
lxc.aa_profile=unconfined
lxc.mount.auto=proc:rw sys:rw cgroup:rw
RAW_LXC
  lxc launch \
    --config security.privileged=true \
    --config security.nesting=true \
    --config linux.kernel_modules=ip_tables,ip6_tables,netlink_diag,nf_nat,overlay \
    --config raw.lxc="${raw_lxc}" \
    my-image my-container

When deploying flannel (kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml), I encounter the following error and the flannel pod doesn’t start:

Dec 13 09:41:40 my-container kubelet[772]: E1213 09:41:40.025941     772 pod_workers.go:182] Error syncing pod aaec5413-dfe9-11e7-b8cb-00163e799a0d ("kube-flannel-ds-26nlp_kube-system(aaec5413-dfe9-11e7-b8cb-00163e799a0d)"), skipping: failed to "StartContainer" for "kube-flannel" with RunContainerError: "failed to start container \"028b057e4f3de8891d0cfdff8c0483a3a52d7bdfd6fbfaf7a5b2aa944516995d\": Error response from daemon: oci runtime error: container_linux.go:265: starting container process caused \"process_linux.go:368: container init caused \\\"process_linux.go:340: setting cgroup config for procHooks process caused \\\\\\\"failed to write a *:* rwm to devices.allow: write /sys/fs/cgroup/devices/kubepods/podaaec5413-dfe9-11e7-b8cb-00163e799a0d/028b057e4f3de8891d0cfdff8c0483a3a52d7bdfd6fbfaf7a5b2aa944516995d/devices.allow: operation not permitted\\\\\\\"\\\"\""

What is missing here with regard to privileges?

Running pods doesn’t fail in general (e.g. nginx works). Flannel runs in privileged context.

Docker cannot write to devices.allow
#2

A few more notes:

I use LXD 2.20 out of git.

The problem doesn’t seem to be linked to the container runtime (I tested with vanilla Docker, Ubuntu Docker and cri-o) or flannel in particular.

Any idea?

#3

Your container has the default devices access setup in the devices cgroup, looks like that nested container is trying to get access to device nodes that the container itself isn’t allowed to access, causing the permission error.

I supposed you could add this to your raw.lxc:

lxc.cgroup.devices.allow = a
#4

Thanks @stgraber - after also adding lxc.cap.drop= flannel actually works. The raw lxc config now is:

lxc.aa_profile=unconfined
lxc.mount.auto=proc:rw sys:rw cgroup:rw
lxc.cgroup.devices.allow=a
lxc.cap.drop=
#5

Is there a solution that does not allow access to all devices?

#6

Is this still correct in LXD 4.18? On the command line for instance lxd doesn’t seem to recognize lxc.aa_profile as a configuration item.

#7

lxc.apparmor.profile when dealing with more recent liblxc

#8

This does not appear to work against the op’s error.

sudo lxc launch -c security.nesting=true -c security.privileged=true ubuntu:20.04 $clustername-control-$d -c raw.lxc="
lxc.apparmor.profile=unconfined
lxc.cap.drop=
lxc.mount.auto=proc:rw sys:rw cgroup:rw"

Error:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: failed to write "a *:* rwm": write /sys/fs/cgroup/devices/docker/6e418cf63e744a0a77edd57a8fe34a4e227844a6a950fe10a30e6cabae5227aa/devices.allow: operation not permitted: unknown.
1 Like
#9

Hi i am having the same error did you managed to deal with it ?