Unable to install docker.io (ubuntu repo) on privileged lxd container (v3.4)

laymonk · September 12, 2018, 11:21pm

Hi,

I needed to install a docker based application that was failing in lxd due to privileged status …

So, I set raw.lxc as follows:

raw.lxc: |-
    lxc.apparmor.profile = unconfined
    lxc.cgroup.devices.allow = a
    lxc.mount.auto = proc:rw sys:rw cgroup:rw
    lxc.cap.drop = mac_override sys_time sys_module sys_rawio

Additionally, I also set the following container security params, and restarted the container

lxc config set mail security.nesting true
lxc config set mail security.privileged true

But to my surprise, I have since been unable to install both docker-ce (from the Docker repo) or docker.io (from the ubuntu repo). Install always fails when docker tries to start during package configuration.

If I remove the raw.lxc settings, docker starts up once again … but then I am unable to install the application. I get exactly the same error whether I try a container based on ubuntu:18.04 or ubuntu:16.04

The errors docker throws up during install (ubuntu:16.04 container)

● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2018-09-12 22:53:10 UTC; 7ms ago
     Docs: https://docs.docker.com
  Process: 1958 ExecStart=/usr/bin/dockerd -H fd:// $DOCKER_OPTS (code=exited, status=1/FAILURE)
 Main PID: 1958 (code=exited, status=1/FAILURE)

Sep 12 22:53:09 mail1 systemd[1]: Starting Docker Application Container Engine...
Sep 12 22:53:09 mail1 dockerd[1958]: time="2018-09-12T23:53:09.363293767+01:00" level=info msg="libcontainerd: new ... 1980"
Sep 12 22:53:10 mail1 dockerd[1958]: time="2018-09-12T23:53:10.379220864+01:00" level=error msg="'overlay' not foun...aded."
Sep 12 22:53:10 mail1 dockerd[1958]: time="2018-09-12T23:53:10.381226073+01:00" level=error msg="'overlay' not foun...aded."
Sep 12 22:53:10 mail1 dockerd[1958]: time="2018-09-12T23:53:10.383325671+01:00" level=error msg="There are no more ...able."
Sep 12 22:53:10 mail1 dockerd[1958]: Error starting daemon: error initializing graphdriver: loopback attach failed
Sep 12 22:53:10 mail1 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Sep 12 22:53:10 mail1 systemd[1]: Failed to start Docker Application Container Engine.
Sep 12 22:53:10 mail1 systemd[1]: docker.service: Unit entered failed state.
Sep 12 22:53:10 mail1 systemd[1]: docker.service: Failed with result 'exit-code'.
Hint: Some lines were ellipsized, use -l to show in full.

journalctl -xe does not show any more useful error

The errors docker throws up during install (ubuntu:18.04 container) - via journalctl -xe

-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit docker.socket has finished shutting down.
Sep 12 22:52:32 mail systemd[1]: Stopping Docker Socket for the API.
-- Subject: Unit docker.socket has begun shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit docker.socket has begun shutting down.
Sep 12 22:52:32 mail systemd[1]: Starting Docker Socket for the API.
-- Subject: Unit docker.socket has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit docker.socket has begun starting up.
Sep 12 22:52:32 mail systemd[1]: Listening on Docker Socket for the API.
-- Subject: Unit docker.socket has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit docker.socket has finished starting up.
--
-- The start-up result is RESULT.
Sep 12 22:52:32 mail systemd[1]: docker.service: Start request repeated too quickly.
Sep 12 22:52:32 mail systemd[1]: docker.service: Failed with result 'exit-code'.
Sep 12 22:52:32 mail systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit docker.service has failed.
--
-- The result is RESULT.
Sep 12 22:52:32 mail systemd[1]: docker.socket: Failed with result 'service-start-limit-hit'.
Sep 12 23:06:00 mail systemd[1]: Starting Cleanup of Temporary Directories...
-- Subject: Unit systemd-tmpfiles-clean.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support

I cannot tell which of the raw.lxc settings is tripping docker within the container …
Could this be something to do with /dev/net/tun … as mentioned in this launchpad bug ?

laymonk · September 13, 2018, 2:01am

I dug a little bit more, and it seems the problem with docker stems from storage drivers … on both ubuntu:18.04 and ubuntu:16.04 …

In syslog, I see the following:

#ubuntu 18.04

failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd

#ubuntu 16.04
'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded.

So, bottom line is that it seems docker is trying to use storage drivers that are not working inside the container … so, overlay in ubuntu 16.04, and btrfs in ubuntu 18.04 …

A few more searching around … and I see (although my issue is not about vfs driver, I think it’s related):

This is Docker’s page on how storage drivers are chosen:

Is there anything I can set in lxd profile for the container that could help ?

laymonk · September 13, 2018, 2:25am

Yes, it seems docker works well with aufs on zfs backing store (which is what I have) …
and setting the following (in lxd profile), restarting the containers and purging/re-installing docker fixed the problem for both ubuntu:18.04 and 16.04

linux.kernel_modules: aufs

512YB · April 27, 2021, 10:32pm

I have similar problem:

systemctl status docker --lines 50
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2021-04-27 22:28:50 UTC; 54s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
    Process: 943 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
   Main PID: 943 (code=exited, status=1/FAILURE)
Apr 27 22:28:50 s1-n2-k1 systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Apr 27 22:28:50 s1-n2-k1 systemd[1]: Stopped Docker Application Container Engine.
Apr 27 22:28:50 s1-n2-k1 systemd[1]: docker.service: Start request repeated too quickly.
Apr 27 22:28:50 s1-n2-k1 systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 27 22:28:50 s1-n2-k1 systemd[1]: Failed to start Docker Application Container Engine.
Apr 27 22:29:01 s1-n2-k1 systemd[1]: docker.service: Start request repeated too quickly.
Apr 27 22:29:01 s1-n2-k1 systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 27 22:29:01 s1-n2-k1 systemd[1]: Failed to start Docker Application Container Engine.

All mentioned above didn’t help me.

UPD: This helped:

zfs create -V 20GB my-pool/docker/blah
mkfs.btrfs /dev/zvol/my-pool/docker/blah
lxc config device add my-container docker disk source=/dev/zvol/my-pool/docker/blah path=/var/lib/docker

(from https://github.com/lxc/lxd/issues/2305)
Hope the posibility to work with zfs inside LXD will be added soon, but untill it done, simplier to use btrfs to run Docker-in-LXD.