Not able to run Kubernetes inside docker inside LXD

I know it is a long-shot to ask the question but it would be great if there would be some explanation from LXD’s point of view. I am trying to get https://github.com/kubernetes-sigs/kind running in an LXD container. Note that I have raised an issue on the project here: https://github.com/kubernetes-sigs/kind/issues/455

This is the configuration on the LXD container:

$ lxc config show tmp --expanded
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 18.04 LTS amd64 (release) (20190402)
  image.label: release
  image.os: ubuntu
  image.release: bionic
  image.serial: "20190402"
  image.version: "18.04"
  raw.idmap: both 1000 1000
  raw.lxc: |
    lxc.cgroup.devices.allow = a
    lxc.cap.drop =
  security.nesting: "true"
  security.privileged: "true"
  volatile.base_image: 663f6663aed66a22dd708c4b07514748221522b810008c55002fcc1dd81af377
  volatile.eth0.hwaddr: 00:16:3e:3d:62:a3
  volatile.idmap.base: "0"
  volatile.idmap.current: '[]'
  volatile.idmap.next: '[]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
devices:
  docker_dir:
    path: /var/lib/docker
    source: /mnt/docker
    type: disk
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

I can for instance run containers with priviliged mode in that container. Example,

d run -it --privileged --rm ubuntu
root@32d0f59bf9c0:/#

However, when I run kind to create a cluster, I get some errors.

Apr 26 12:22:55 kind-control-plane kubelet[4908]: I0426 12:22:55.917788    4908 server.go:999] Started kubelet
Apr 26 12:22:55 kind-control-plane kubelet[4908]: I0426 12:22:55.917841    4908 server.go:137] Starting to listen on 0.0.0.0:10250
Apr 26 12:22:55 kind-control-plane kubelet[4908]: E0426 12:22:55.918090    4908 event.go:212] Unable to write event: 'Post https://172.17.0.2:6443/api/v1/namespaces/default/events: dial tcp 172.17.0.2:6443: connec
t: connection refused' (may retry after sleeping)
Apr 26 12:22:55 kind-control-plane kubelet[4908]: I0426 12:22:55.918328    4908 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
Apr 26 12:22:55 kind-control-plane kubelet[4908]: I0426 12:22:55.918350    4908 status_manager.go:152] Starting to sync pod status with apiserver
Apr 26 12:22:55 kind-control-plane kubelet[4908]: I0426 12:22:55.918371    4908 kubelet.go:1829] Starting kubelet main sync loop.
Apr 26 12:22:55 kind-control-plane kubelet[4908]: I0426 12:22:55.918390    4908 kubelet.go:1846] skipping pod synchronization - [container runtime status check may not have completed yet PLEG is not healthy: pleg 
has yet to be successful]
Apr 26 12:22:55 kind-control-plane kubelet[4908]: I0426 12:22:55.918473    4908 server.go:333] Adding debug handlers to kubelet server.
Apr 26 12:22:55 kind-control-plane kubelet[4908]: I0426 12:22:55.918475    4908 volume_manager.go:248] Starting Kubelet Volume Manager
Apr 26 12:22:55 kind-control-plane kubelet[4908]: I0426 12:22:55.918497    4908 desired_state_of_world_populator.go:130] Desired state populator starts to run
Apr 26 12:22:55 kind-control-plane kubelet[4908]: W0426 12:22:55.921436    4908 cni.go:203] Unable to update cni config: No networks found in /etc/cni/net.d
Apr 26 12:22:55 kind-control-plane kubelet[4908]: E0426 12:22:55.925886    4908 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin 
is not ready: cni config uninitialized
Apr 26 12:22:55 kind-control-plane kubelet[4908]: W0426 12:22:55.935038    4908 manager.go:349] Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: no such file or directory
Apr 26 12:22:55 kind-control-plane kubelet[4908]: I0426 12:22:55.950959    4908 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Apr 26 12:22:55 kind-control-plane kubelet[4908]: I0426 12:22:55.952644    4908 cpu_manager.go:155] [cpumanager] starting with none policy
Apr 26 12:22:55 kind-control-plane kubelet[4908]: I0426 12:22:55.952660    4908 cpu_manager.go:156] [cpumanager] reconciling every 10s
Apr 26 12:22:55 kind-control-plane kubelet[4908]: I0426 12:22:55.952670    4908 policy_none.go:42] [cpumanager] none policy: Start
Apr 26 12:22:55 kind-control-plane kubelet[4908]: F0426 12:22:55.953144    4908 kubelet.go:1384] Failed to start ContainerManager [open /proc/sys/vm/overcommit_memory: permission denied, open /proc/sys/kernel/pani
c: permission denied, open /proc/sys/kernel/panic_on_oops: permission denied]
Apr 26 12:22:55 kind-control-plane systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Apr 26 12:22:55 kind-control-plane systemd[1]: kubelet.service: Failed with result 'exit-code'.

Notably, the permission denied on
/proc/sys/vm/overcommit_memory /proc/sys/kernel/panic /proc/sys/kernel/panic_on_oops as well as this one open /dev/kmsg: no such file or directory

Could there be some missing configuration on LXD’s side which is causing the reported errors above? Thanks in advance.

It’s a bit of a pig getting it to work in LXD but it is possible.

I used ubuntu 18.04 as the lxd OS
16.04 also works

Docker CE was version 18.06.0 ce 3 on 18.04
Docker CE was 17.06 on 18.04

Here is my profile for 16.04:

### This is a yaml representation of the configuration.
### Any line starting with a '# will be ignored.
###
### A sample configuration looks like:
### name: container1
### profiles:
### - default
### config:
###   volatile.eth0.hwaddr: 00:16:3e:e9:f8:7f
### devices:
###   homedir:
###     path: /extra
###     source: /home/user
###     type: disk
### ephemeral: false
###
### Note that the name is shown but cannot be changed
 
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 16.04 LTS amd64 (release) (20181004)
  image.label: release
  image.os: ubuntu
  image.release: xenial
  image.serial: "20181004"
  image.version: "16.04"
  linux.kernel_modules: openvswitch,nbd,ip_tables,ip6_tables,netlink_diag,nf_nat,overlay
  raw.lxc: |
lxc.apparmor.profile=unconfined
lxc.mount.auto=proc:rw sys:rw cgroup:rw
lxc.cap.drop=
lxc.cgroup.devices.allow=a
  security.nesting: "true"
  security.privileged: "true"
  volatile.base_image: c966933fdfd390d301fed3447528e2f910bf72c0615b2caaf3235a791fed3541
  volatile.eth0.hwaddr: 00:16:3e:46:5f:f7
  volatile.idmap.base: "0"
  volatile.idmap.next: '[]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.lxdbr1.hwaddr: 00:16:3e:9d:64:5a
  volatile.lxdbr1.name: eth1
devices:
  aadisable:
path: /sys/module/nf_conntrack/parameters/hashsize
source: /dev/null
type: disk
  aadisable1:
path: /sys/module/apparmor/parameters/enabled
source: /dev/null
type: disk
  mem:
path: /dev/mem
type: unix-char

Also have to make sure you enable swapoff -a

and I remember I had to ignore preflight errors with kubadm init

kubeadm init --ignore-preflight-errors=all

The following config works for me to run Kubernetes in LXD (albeit with cri-o): https://github.com/schu/kubedee/blob/c40324e143d768edcfbef1ddc2d4c202fe60164b/lib.bash#L1117-L1132

Note that I also bind mount /dev/kmsg: https://github.com/schu/kubedee/blob/c40324e143d768edcfbef1ddc2d4c202fe60164b/lib.bash#L1168-L1172

1 Like