LXD Containers and Virtual Machines Can't Run Docker --privileged

LXD
1

Hi All,

First I just want to say thank you to everyone here, especially Stefan for all of your diligent work on both LXD as well as supporting it for the rest of us. I’ve found this site to be extremely important to understanding and working with LXD.

I have set my default profile as such:
lxc profile show default
config:
limits.memory.swap: “false”
linux.kernel_modules: overlay,nf_nat,ip_tables,ip6_tables,netlink_diag,br_netfilter,xt_conntrack,nf_conntrack,ip_vs,vxlan
raw.lxc: “lxc.apparmor.profile=unconfined\nlxc.cap.drop= \nlxc.cgroup.devices.allow=a\nlxc.mount.auto=proc:rw
sys:rw\n”
security.nesting: “true”
security.privileged: “true”
description: Default LXD profile
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: local
type: disk
name: default
used_by:
- /1.0/instances/jjtest-cluster-master

I’m having problems running docker containers in privileged mode inside lxd containers AND Virtual Machines. The latter was a bit of a surprise to me as I thought perhaps there was just an issue with containers within containers.
For Instance, I can run this command in a VMWare VM without an issue

sudo docker run -d --restart=unless-stopped -p 80:80 -p 443:443 --privileged rancher/rancher

The service starts without an issue.

If I run it in a lxd container or in a virtual machine I get:

> ERROR: Rancher must be ran with the --privileged flag when running outside of Kubernetes

I’ll admit I’ve had issues with formatting the appropriate profile parameters and had a lot of confusion between the documentation for various versions of LXD as it pertains to raw.lxc.

I’m running lxd 4.18 on Ubuntu 20.04 latest patching applied.

If someone could guide me on this issue, I’d be very grateful…

Best,

Joel

2

Please can you show lxc config show <instance> for both the container and VM you’re using.

3

Virtual Machine
architecture: x86_64
config:
image.architecture: amd64
image.description: Ubuntu focal amd64 (20210926_07:42)
image.os: Ubuntu
image.release: focal
image.serial: “20210926_07:42”
image.type: disk-kvm.img
image.variant: default
volatile.base_image: 696653a2a470f00aa3e6a0ac5f7ff92300b14bafeb5de01ec1a2f855d712b822
volatile.eth0.host_name: tapa4051c7e
volatile.eth0.hwaddr: 00:16:3e:0a:44:83
volatile.last_state.power: RUNNING
volatile.uuid: 9f7f5d0e-1266-43df-a7c4-14d76d88c23a
volatile.vsock_id: “228”
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: “”

Container
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 20.04 LTS amd64 (release) (20210921)
image.label: release
image.os: ubuntu
image.release: focal
image.serial: “20210921”
image.type: squashfs
image.version: “20.04”
security.nesting: “true”
security.privileged: “true”
volatile.base_image: a068e8daef0f88c667fd0f201ed0de1c48693ee383eeafbee6a51b79b0d29fea
volatile.eth0.host_name: veth7432ceb3
volatile.eth0.hwaddr: 00:16:3e:68:bf:5e
volatile.idmap.base: “0”
volatile.idmap.current: ‘[]’
volatile.idmap.next: ‘[]’
volatile.last_state.idmap: ‘[]’
volatile.last_state.power: RUNNING
volatile.uuid: eb8fb350-43b9-459d-9d34-5f438285d9d0
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: “”

4

Do you know what that error means? What is it checking for?

Running inside a LXD VM runs a full standard kernel separate from the host (so AppArmor and raw.lxc do not apply from the host), but it looks like thats not what this command is checking for.

5

@tomp -
I can’t confirm 100% but it would appear that it’s trying to create a bridged interface and a new subnet (assumably for transport communication.) Not finding much in the logs sadly.

Here’s a snippet of what repeats on the parent vm/container with every docker container restart.

Sep 27 16:50:57 jjtest-cluster-master containerd[2156]: time=“2021-09-27T16:50:57.656862272Z” level=error msg=“copy shim log” error=“read /proc/self/fd/12: file already closed”
Sep 27 16:50:57 jjtest-cluster-master systemd[1392]: var-lib-docker-overlay2-3aefae99b3f412e0fae75ea911c1cd407d06b0d5e64e11ba700942a69807cc4b-merged.mount: Succeeded.
Sep 27 16:50:57 jjtest-cluster-master systemd[1]: var-lib-docker-overlay2-3aefae99b3f412e0fae75ea911c1cd407d06b0d5e64e11ba700942a69807cc4b-merged.mount: Succeeded.
Sep 27 16:50:57 jjtest-cluster-master containerd[2156]: time=“2021-09-27T16:50:57.922295783Z” level=info msg=“starting signal loop” namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/fde4ed286eb1354fb93d01478f990a898b486262dcf1fb9f55ddb0a710e06596 pid=2840
Sep 27 16:50:57 jjtest-cluster-master systemd[1392]: run-docker-runtime\x2drunc-moby-fde4ed286eb1354fb93d01478f990a898b486262dcf1fb9f55ddb0a710e06596-runc.1CQcoh.mount: Succeeded.
Sep 27 16:50:57 jjtest-cluster-master systemd[1]: run-docker-runtime\x2drunc-moby-fde4ed286eb1354fb93d01478f990a898b486262dcf1fb9f55ddb0a710e06596-runc.1CQcoh.mount: Succeeded.
Sep 27 16:51:00 jjtest-cluster-master dockerd[2289]: time=“2021-09-27T16:51:00.038106647Z” level=info msg=“ignoring event” container=fde4ed286eb1354fb93d01478f990a898b486262dcf1fb9f55ddb0a710e06596 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Sep 27 16:51:00 jjtest-cluster-master containerd[2156]: time=“2021-09-27T16:51:00.043619689Z” level=info msg=“shim disconnected” id=fde4ed286eb1354fb93d01478f990a898b486262dcf1fb9f55ddb0a710e06596
Sep 27 16:51:00 jjtest-cluster-master containerd[2156]: time=“2021-09-27T16:51:00.044149234Z” level=error msg=“copy shim log” error=“read /proc/self/fd/12: file already closed”
Sep 27 16:51:00 jjtest-cluster-master systemd[1392]: var-lib-docker-overlay2-3aefae99b3f412e0fae75ea911c1cd407d06b0d5e64e11ba700942a69807cc4b-merged.mount: Succeeded.
Sep 27 16:51:00 jjtest-cluster-master systemd[1]: var-lib-docker-overlay2-3aefae99b3f412e0fae75ea911c1cd407d06b0d5e64e11ba700942a69807cc4b-merged.mount: Succeeded.
Sep 27 16:51:00 jjtest-cluster-master containerd[2156]: time=“2021-09-27T16:51:00.493377789Z” level=info msg=“starting signal loop” namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/fde4ed286eb1354fb93d01478f990a898b486262dcf1fb9f55ddb0a710e06596 pid=2937
Sep 27 16:51:00 jjtest-cluster-master systemd[1392]: run-docker-runtime\x2drunc-moby-fde4ed286eb1354fb93d01478f990a898b486262dcf1fb9f55ddb0a710e06596-runc.9wfBXw.mount: Succeeded.
Sep 27 16:51:00 jjtest-cluster-master systemd[1]: run-docker-runtime\x2drunc-moby-fde4ed286eb1354fb93d01478f990a898b486262dcf1fb9f55ddb0a710e06596-runc.9wfBXw.mount: Succeeded.
Sep 27 16:51:02 jjtest-cluster-master dockerd[2289]: time=“2021-09-27T16:51:02.594058249Z” level=info msg=“ignoring event” container=fde4ed286eb1354fb93d01478f990a898b486262dcf1fb9f55ddb0a710e06596 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Sep 27 16:51:02 jjtest-cluster-master containerd[2156]: time=“2021-09-27T16:51:02.595977586Z” level=info msg=“shim disconnected” id=fde4ed286eb1354fb93d01478f990a898b486262dcf1fb9f55ddb0a710e06596

6

For instance on the VMWare VM I see this in the logs on-launch

Sep 27 15:23:10 lv-juju-01 kernel: [600098.363489] docker0: port 1(veth7b861ba) entered blocking state
Sep 27 15:23:10 lv-juju-01 kernel: [600098.363494] docker0: port 1(veth7b861ba) entered disabled state
Sep 27 15:23:10 lv-juju-01 kernel: [600098.363677] device veth7b861ba entered promiscuous mode
Sep 27 15:23:10 lv-juju-01 kernel: [600098.364740] IPv6: ADDRCONF(NETDEV_UP): veth7b861ba: link is not ready
Sep 27 15:23:10 lv-juju-01 networkd-dispatcher[1100]: WARNING:Unknown index 5 seen, reloading interface list
Sep 27 15:23:10 lv-juju-01 systemd-networkd[793]: veth7b861ba: Link UP
Sep 27 15:23:10 lv-juju-01 systemd-timesyncd[690]: Network configuration changed, trying to establish connection.
Sep 27 15:23:10 lv-juju-01 systemd-timesyncd[690]: Synchronized to time server IP_ADDRESS:123 (10.192.14.10).
Sep 27 15:23:10 lv-juju-01 systemd-udevd[31085]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Sep 27 15:23:10 lv-juju-01 systemd-udevd[31085]: Could not generate persistent MAC address for veth4d099f4: No such file or directory
Sep 27 15:23:10 lv-juju-01 systemd-timesyncd[690]: Network configuration changed, trying to establish connection.
Sep 27 15:23:10 lv-juju-01 systemd-timesyncd[690]: Synchronized to time server <IP_Address>:123 (<IP_ADDRESS).
Sep 27 15:23:10 lv-juju-01 systemd-timesyncd[690]: Network configuration changed, trying to establish connection.

7

Can you provide me a list of reproducer steps including launching the VM, installing docker, then I can try it. Thanks

8

I ended up giving up on the Rancher Kubernetes deployment against LXD vm/containers. IMHO it should just work and it does not.

1 Like
9

Hm, I just gave a quick spin for this in a LXD ubuntu:20.04 VM (so not in a container), and it works OOTB…? I just installed docker (with a quick “curl -L get.docker.com | sh”) and then just ran rancher as a privileged docker container (and checked logs and GUI).

10

I have the same issue in LXD 4/5… using the --privilege flag and a proper docker profile for lxc/lxd.

Rancher is k8 or k3’s - you may need to use a microk8s profile. it used to work in a standard lxc docker profile but things may have changed.

That may be why it is working with a vm and not a container. try microk8s lxc setup then install docker then try it again. A vm is probably best with long horn.
→
https://microk8s.io/docs/install-lxd

wget https://raw.githubusercontent.com/ubuntu/microk8s/master/tests/lxc/microk8s.profile -O microk8s.profile

Microk8s can also run rancher some examples.

https://www.youtube.com/watch?v=BLK9dqlY0K0 (sxy robot voice included)

Helm install

good luck’
~md