Docker cannot write to devices.allow

I’ve a issue whilst try to run docker containers inside an lxc container(using lxd).

I tried to setup the following docker-compose project:
https://mailcow.github.io/mailcow-dockerized-docs/

When running the "docker-compose up -d " command, the following error appears:

ERROR: for fail2ban-mailcow  Cannot start service fail2ban-mailcow: OCI runtime create failed: container_linux.go:296: starting container process caused "process_linux.go:398: container init caused \"process_linux.go:365: setting cgroup config for procHooks process caused \\\"failed to write a *:* rwm to devices.allow: write /sys/fs/cgroup/devices/docker/9cc4fa5ee410af11cdaaffd1a046979be3a9e0db3beaf53988dfbef8ad614999/devices.allow: operation not permitted\\\"\"": unknown

ERROR: for ipv6nat  Cannot start service ipv6nat: OCI runtime create failed: container_linux.go:296: starting container process caused "process_linux.go:398: container init caused \"process_linux.go:365: setting cgroup config for procHooks process caused \\\"failed to write a *:* rwm to devices.allow: write /sys/fs/cgroup/devices/docker/a23565f8a46dae7e69164f2041793bb3958a273891573ec0028440ffa27598a9/devices.allow: operation not permitted\\\"\"": unknown

The lxc container is running ubuntu-daily:17.10 currently. The error occours on ubuntu:17.10 too.

ubuntu:16.04 and images:ubuntu/xenial fails earlier.

I tried docker via package(docker.io) and current master from git.

After setting a raw.lxc config like here, docker cannot start anymore.
The configuration security.nesting and security.privileged are true.

After setting raw.lxc following error stops docker from working:

Dec 30 19:09:58 mail dockerd[1245]: time="2017-12-30T19:09:58.858154568Z" level=error msg="There are no more loopback devices available."
Dec 30 19:09:58 mail dockerd[1245]: time="2017-12-30T19:09:58.858341501Z" level=error msg="[graphdriver] prior storage driver devicemapper failed: loopback attach failed"
Dec 30 19:09:58 mail dockerd[1245]: Error starting daemon: error initializing graphdriver: loopback attach failed

When i apply linux.kernel_modules from the other thread, nothing is changed.

Hi there! I’m not sure if this helps you out, but I fired up a lxd instance on my system this evening to test the mailcow deployment. It went pretty smoothly, so hopefully my experiences can help you get it deployed similarly.

The entirety of my deployment is at https://asciinema.org/a/yfaU6Gc5aRuASglBC38wtriR3

I had to run a privileged base lxd container to get this working correctly, though, so keep that in mind from a security perspective.

What isn’t shown in that asciinema output, is when I created my docker profile. I did that with lxc profile copy default docker and then edited with lxc profile edit docker adding the raw.lxc configuration as follows:

config:
  raw.lxc: |-
    lxc.aa_profile = unconfined
    lxc.cgroup.devices.allow = a
    lxc.mount.auto=proc:rw sys:rw
    lxc.cap.drop=

When you deploy the lxd instance, make sure you enable the configuration settings that allow Docker to run embedded, which are security.nesting=true and security.privileged=true.

You can do this at lxc launch time with the -c flag or after-the-launch with a combination of:

lxc config set mailcow security.nesting true
lxc config set mailcow security.privileged true
lxc restart mailcow

Good luck, and I hope you manage to get everything working as expected.

Cheers,

  • Andrew

3 Likes

Did you find a solution? I have the same Problem in a XEN vm:

This doesn’t seem to work anymore… Anyone know why?

You have to replace lxc.aa_profile with lxc.apparmor.profile

Confirmed working :slight_smile:

1 Like

Is there a way to make this configuration work but without security.privileged?

I get operation not permitted error on writing to devices.allow file in cgroups.

This is an effort to consolidate the answer for LXD 4.x and Docker (specifically I’m trying to install Rancher on a lxd cluster.) Please let me know if there’s a post/thread that’s answering this specifically, but I’m having issues getting docker happy in the lxd 4 containers.

i have

sudo lxc config set $clustername-control-$d security.syscalls.intercept.setxattr=true security.syscalls.intercept.mknod=true raw.lxc="raw.apparmor = unconfined lxc.cgroup.dev ices.allow = a lxc.mount.auto = proc:rw sys:rw"

and am consistently seeing

docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: failed to write "a *:* rwm": write /sys/fs/cgroup/devices/docker/3edde81bd81feaa8333960eeaa146f3f2ff87820e4ba32a7b82f3a759166a22f/devices.allow: operation not permitted: unknown.

Indicating that my lxd container is indeed not permitting underlying calls. I appreciate any direction you can give. Thanks.

Have you followed the steps in:

https://www.youtube.com/watch?v=_fCSSEyiGro

Also what storage backend are you using?