LXD run docker-ce on centos 7

LXD
1

Hi,

I can’t run docker-ce inside centos lxd container. Versions are :

Host where lxd is installed: CentOS Linux release 7.5.1804 (Core)
Container LXC/LXD: CentOS Linux release 7.6. 1810 (Core)
Docker-ce installed inside LXC:18.09.0

Config of the container are :

config:
image.architecture: amd64
image.description: Centos 7 amd64 (20181221_02:16)
image.os: Centos
image.release: “7”
image.serial: “20181221_02:16”
linux.kernel_modules: bridge,br_netfilter,ip_tables,ip6_tables,ip_vs,netlink_diag,nf_nat,overlay,xt_conntrack
raw.lxc: |-
lxc.apparmor.profile = unconfined
lxc.cgroup.devices.allow = a
lxc.cap.drop =
lxc.mount.auto = proc:rw sys:rw cgroup:rw:force cgroup-full:rw:force
security.nesting: “true”
security.privileged: “true”

Here the systemd logs of the failure.

systemd: Starting Docker Application Container Engine…
dockerd: level=info msg=“parsed scheme: “unix”” module=grpc
dockerd: level=info msg=“scheme “unix” not registered, fallback to default scheme” module=grpc
dockerd: level=info msg=“parsed scheme: “unix”” module=grpc
dockerd: level=info msg=“scheme “unix” not registered, fallback to default scheme” module=grpc
dockerd: level=info msg=“ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0 }]” module=grpc
dockerd: level=info msg=“ClientConn switching balancer to “pick_first”” module=grpc
dockerd: level=info msg=“pickfirstBalancer: HandleSubConnStateChange: 0xc42076f7e0, CONNECTING” module=grpc
dockerd: level=info msg=“pickfirstBalancer: HandleSubConnStateChange: 0xc42076f7e0, READY” module=grpc
dockerd: level=info msg=“ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0 }]” module=grpc
dockerd: level=info msg=“ClientConn switching balancer to “pick_first”” module=grpc
dockerd: level=info msg=“pickfirstBalancer: HandleSubConnStateChange: 0xc42076fab0, CONNECTING” module=grpc
dockerd: level=info msg=“pickfirstBalancer: HandleSubConnStateChange: 0xc42076fab0, READY” module=grpc
dockerd: level=info msg=“Graph migration to content-addressability took 0.00 seconds”
dockerd: level=warning msg=“Your kernel does not support cgroup memory limit”
dockerd: level=warning msg=“Unable to find blkio cgroup in mounts”
dockerd: level=warning msg=“Unable to find cpuset cgroup in mounts”
dockerd: level=warning msg=“mountpoint for pids not found”
dockerd: Error starting daemon: Devices cgroup isn’t mounted
systemd: docker.service: main process exited, code=exited, status=1/FAILURE
systemd: Failed to start Docker Application Container Engine.
systemd: Unit docker.service entered failed state.
systemd: docker.service failed.

Have any one succeeded working with Docker-ce with those versions of LXD / Docker-ce / CentOs ?

Thanks

2 
dockerd: Error starting daemon: Devices cgroup isn’t mounted

So docker isn’t happy about the cgroup layout in your container, I suspect your direct interference with lxc.mount.auto isn’t helping here as that would effectively turn off cgroup namespacing that’s needed in this case.

3

This appears to be the error.
It is probably specific to Centos because I can run docker-ce on an Ubuntu host.

5

Message deleted for being unable to make docker-ce 18.09 works again with lxc centos/7

6

Hi, it says that the last post was withdrawn by the author.
Have you managed to make docker-ce work on centos7?

7

Hi,

Unfortunately, I was not able to make Docker-ce working within LXD container like I wanted. Since cgroups are messed up when the host is Centos. This true whatever if the container is centos or ubuntu.

However, I was able to make docker-ce work with a centos guest from an ubuntu host. Which it is not my targeted platform.

Following, a synthesis of my tests with LXD 3.10

Host            | Guest         | Docker-ce 18.09
-------------------------------------------------
Centos 7.5      | Centos 7.6    | Not working
Centos 7.5      | Ubuntu        | Not working 
Ubuntu          | Centos 7.6    | OK
Ubuntu          | Ubuntu        | OK

To make Docker-ce works in a Centos container from a Centos host, i had to remove LXD and work with the native LXC packages. Starting with :

snap remove lxd
yum install lxc lxc-templates lxc-extra

I hope that LXD will be patched soon for cgroup mounts from centos hosts.

Regards,

8

LXD can’t really do much unfortunately. The kernel requires the cgroup layout of the containers to match that of the host which is what you’re seeing here.

What do you see on your host when you do:

  • cat /proc/self/cgroup
  • cat /proc/cgroups
  • grep cgroup /proc/self/mountinfo
9

lxc-create manage to create centos container with “correct” cgroup. I was pointing out that the only variable on my tests is the daemon behind those commands “lxc-create” or “lxc launch”. Since “lxc launch” doesn’t provide viable cgrouped centos container I though that the LXD software was bugged. I am maybe wrong and i may miss a third party that i am not aware of.

From the host i got thoses results:

$ cat /proc/self/cgroup
11:memory:/user.slice
10:freezer:/
9:pids:/user.slice
8:perf_event:/
7:cpuset:/
6:devices:/user.slice
5:hugetlb:/
4:blkio:/user.slice
3:net_prio,net_cls:/
2:cpuacct,cpu:/user.slice
1:name=systemd:/user.slice/user-1000.slice/session-6.scope

.

$ cat /proc/cgroups
#subsys_name    hierarchy       num_cgroups     enabled
cpuset  7       10      1
cpu     2       75      1
cpuacct 2       75      1
memory  11      75      1
devices 6       75      1
freezer 10      10      1
net_cls 3       10      1
blkio   4       75      1
perf_event      8       10      1
hugetlb 5       10      1
pids    9       75      1
net_prio        3       10      1

.

$ grep cgroup /proc/self/mountinfo
24 17 0:20 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:8 - tmpfs tmpfs ro,seclabel,mode=755
25 24 0:21 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:9 - cgroup cgroup rw,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
27 24 0:23 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:10 - cgroup cgroup rw,seclabel,cpuacct,cpu
28 24 0:24 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:11 - cgroup cgroup rw,seclabel,net_prio,net_cls
29 24 0:25 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:12 - cgroup cgroup rw,seclabel,blkio
30 24 0:26 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:13 - cgroup cgroup rw,seclabel,hugetlb
31 24 0:27 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:14 - cgroup cgroup rw,seclabel,devices
32 24 0:28 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:15 - cgroup cgroup rw,seclabel,cpuset,clone_children
33 24 0:29 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:16 - cgroup cgroup rw,seclabel,perf_event
34 24 0:30 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,seclabel,pids
35 24 0:31 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:18 - cgroup cgroup rw,seclabel,freezer
36 24 0:32 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:19 - cgroup cgroup rw,seclabel,memory

Regards,

10

I have the exact same problem on RHEL 7.6, docker will not work when creating Ubuntu 16.04 with lxd 3.0.3 or 3.13, while when creating Ubuntu with lxc-create Docker works like a charm.

11

Hi, any chance the new 3.15 version take care the centos docker issue?

12

HI,

I can tell that lxd version 4.4 on centos 8.2.2004 is working fine with images centos/8/amd64 & centos/7/amd64 for docker-ce 18.09.1 :champagne:

Still, custom config is needed as follow:

lxc exec -n NAME -- sh -c "mkdir -p /lib/modules/<kernel>/"
lxc file push /lib/modules/<kernel>/modules.dep.bin NAME/lib/modules/<kernel>/

lxc config set NAME linux.kernel_modules bridge,br_netfilter,ip_tables,ip6_tables,ip_vs,netlink_diag,nf_nat,overlay,xt_conntrack
lxc config set NAME security.nesting true
lxc config set NAME security.privileged true
lxc config set NAME raw.lxc 'lxc.apparmor.profile = unconfined
lxc.cgroup.devices.allow = a
lxc.cap.drop ='

Best regards