Nested LXD containers with Docker cannot run containers

LXD
1

Hello,

I am trying to run kubernetes inside an LXD container (c1) with nested LXD container (c1.1). I have been able to install Docker-CE into c1.1. As a test, i decided to run a docker container inside in c1.1 but, that failed:

$ docker run hello-world
docker: Error response from daemon: Could not check if docker-default AppArmor profile was loaded: open /sys/kernel/security/apparmor/profiles: permission denied. 
ERRO[0000] error waiting for container: context canceled

The configuration on c1.1 is as follows:

architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 18.10 amd64 (release) (20190212)
  image.label: release
  image.os: ubuntu
  image.release: cosmic
  image.serial: "20190212"
  image.version: "18.10"
  linux.kernel_modules: ip_tables,ip6_tables,nf_nat
  raw.lxc: "lxc.apparmor.profile=unconfined\nlxc.cap.drop= \nlxc.cgroup.devices.allow=a\nlxc.mount.auto=proc:rw sys:rw"
  security.nesting: "true"
  volatile.base_image: 2d62fe42c0971ea71a525c07de4d8aaf53ccd31fceee74312d5327502c10b36f
  volatile.eth0.hwaddr: 00:16:3e:da:32:04
  volatile.idmap.base: "0"
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":999900000},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":999900000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":999900000},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":999900000}]'
  volatile.last_state.power: STOPPED
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

Inside c1.1 i looked at the messages on the docker service and I see some issues:

Mar 04 19:13:08 km dockerd[163]: time="2019-03-04T19:13:08.038776304Z" level=info msg="Docker daemon" commit=774a1f4 graphdriver(s)=vfs version=18.09.3
Mar 04 19:13:08 km dockerd[163]: time="2019-03-04T19:13:08.038862907Z" level=info msg="Daemon has completed initialization"
Mar 04 19:13:08 km dockerd[163]: time="2019-03-04T19:13:08.045496750Z" level=info msg="API listen on /var/run/docker.sock"
Mar 04 19:13:08 km systemd[1]: Started Docker Application Container Engine.
Mar 04 19:13:09 km systemd[1]: docker.service: Failed to reset devices.list: Operation not permitted
Mar 04 19:13:10 km dockerd[163]: time="2019-03-04T19:13:10.549752014Z" level=error msg="a6afd902ebff988429887cb60e4f0a9520758a7ead206f25ca2c0f8937e8d804 cleanup: failed to delet
Mar 04 19:13:10 km dockerd[163]: time="2019-03-04T19:13:10.549788814Z" level=error msg="Handler for POST /v1.39/containers/a6afd902ebff988429887cb60e4f0a9520758a7ead206f25ca2c0f
Mar 04 19:13:11 km systemd[1]: docker.service: Failed to reset devices.list: Operation not permitted
Mar 04 19:18:13 km systemd[1]: docker.service: Failed to reset devices.list: Operation not permitted
Mar 04 19:18:14 km systemd[1]: docker.service: Failed to reset devices.list: Operation not permitted

The logs on c1.1 shows that some properties cannot be alloted:

lxc km 20190304191305.314 WARN     conf - conf.c:lxc_map_ids:2970 - newuidmap binary is missing
lxc km 20190304191305.318 WARN     conf - conf.c:lxc_map_ids:2976 - newgidmap binary is missing
lxc km 20190304191305.325 WARN     conf - conf.c:lxc_map_ids:2970 - newuidmap binary is missing
lxc km 20190304191305.328 WARN     conf - conf.c:lxc_map_ids:2976 - newgidmap binary is missing
lxc km 20190304191305.625 WARN     conf - conf.c:lxc_setup_devpts:1641 - Invalid argument - Failed to unmount old devpts instance
lxc km 20190304191305.626 WARN     cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2406 - Failed to set "devices.allow" to "a"

I am wondering what kind of permissions i must set on the nested containers in order to make them worthy woker nodes for kubernetes. Thanks in advance if there are any ideas?

2

After some more tweaking and testing, it would appear that to have docker run successfully executed inside c1.1, would be to enable security.privileged on it. I have no idea how to make it work with unprivileged containers. Any ideas would be most welcome.

Furthermore, while exploring the issue, i notice that in c1, modprobe commands fail on c1. For instance:

$ modprobe ip_tables
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.18.0-15-generic/modules.dep.bin'                                      
modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.18.0-15-generic

In fact the content of the folder /lib/modules/ is empty. I had added linux.kernel_modules: ip_tables on the container c1 but, that does not help either. Any ideas here as well would be most welcoming.

3

Looks like somebody else had a similar problem solved by setting security.nesting=true:

Edit: Oh, it looks like that is already in your configuration from your post above. :confused:

4

I think this relates to nesting and how many levels of nesting you can have.
Can you have more than one level of nesting in either Docker or LXC/LXD?