How to mount the docker.sock file to LXC/ LXD container?

lxd
snap

(Hagen Kuehn) #1

We are using LXD, among others, to have a local development environment that can be shared between developers. Without a fuse, we can run Docker completely isolated within the LXD/ LXC container. That’s great! Those LXD/ LXC containers are short lived and used for only one development task.

In order to reduce network load and traffic charges, we would like to “run” Docker within the LXD/ LXC container and actually use the Docker instance running on the host machine (typically its run on a developer’s laptop that runs Debian/ Ubuntu). This way the Docker images (or slices thereof) don’t need to be downloaded again since executing Docker commands within the LXD/ LXC container are actually using the host’s Docker. Basically, we want the equivalent of docker run -v /var/run/docker.sock:/var/run/docker.sock ... in LXD/ LXC.

We tried the below without success:

  1. Created LXD/ LXC container (still stopped)
  2. Added device with either one of the below commands
lxc config device add container-name dockerbladevice disk path=/var/run/docker.sock 
source=/var/lib/snapd/hostfs/var/run/docker.sock

lxc config device add container-name dockerbladevice disk path=/var/run/docker.sock 
source=/var/run/docker.sock
  1. Results in error
$ lxc start container-name
Error: Missing source '/var/lib/snapd/hostfs/var/run/docker.sock' for disk 'dockerbladevice'
Try `lxc info --show-log container-name` for more info
  1. Info output
$ lxc info --show-log container-name                                              
Name: container-name
Remote: unix://
Architecture: x86_64
Created: 2018/04/27 10:26 UTC
Status: Stopped
Type: persistent
Profiles: default

Log:

lxc 20180427102823.671 WARN     lxc_conf - conf.c:lxc_map_ids:2831 - newuidmap binary is missing
lxc 20180427102823.671 WARN     lxc_conf - conf.c:lxc_map_ids:2837 - newgidmap binary is missing
lxc 20180427102823.685 WARN     lxc_conf - conf.c:lxc_map_ids:2831 - newuidmap binary is missing
lxc 20180427102823.686 WARN     lxc_conf - conf.c:lxc_map_ids:2837 - newgidmap binary is missing
lxc 20180427102823.688 WARN     lxc_cgfsng - cgroups/cgfsng.c:chowmod:1881 - No such file or directory - 
Failed to chown(/sys/fs/cgroup/unified//lxc/container-name/cgroup.threads, 1000000000, 0)
lxc 20180427103203.468 WARN     lxc_conf - conf.c:lxc_map_ids:2831 - newuidmap binary is missing
lxc 20180427103203.468 WARN     lxc_conf - conf.c:lxc_map_ids:2837 - newgidmap binary is missing
$ lxc config show container-name
architecture: x86_64
config:
 image.description: Quater Controller
 volatile.base_image: fcb2c0c480d0c086c9c87fe0a7f7399b882d5096e862e33a9a8c88648102b431
 volatile.eth0.hwaddr: 00:16:3e:42:75:d2
 volatile.eth0.name: eth0
 volatile.idmap.base: "0"
 volatile.idmap.next: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
 volatile.last_state.idmap: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
 volatile.last_state.power: STOPPED
devices:
  dockerbladevice:
    path: /var/run/docker.sock
    source: /var/run/docker.sock
    type: disk
ephemeral: false
profiles:
  • default
    stateful: false
    description: “”

Environment details:

  LXD installed with Snap
  LXD version: 3.0.0

  driver: lxc
  driver_version: 3.0.0
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.13.0-39-generic
  server: lxd
  server_pid: 2255
  server_version: 3.0.0
  storage: btrfs
  storage_version: "4.4"
  server_clustered: false

How to mount the LXD socket from the host to a LXD container?
How to mount the LXD socket from the host to a LXD container?
(Christian Brauner) #2

This looks like a path-based issue with snapd. That’s something @stgraber can surely help with. Because - at least with a privileged container - this works fine for me. I don’t expect it to work for an unprivileged container because sockets have a owning user namespace which in this case will be the initial user namespace but an unprivileged container will have a separate user namespace. So the kernel will check your permissions against the owning user namespace of the socket (which is the initial user namespace) but your container won’t hold any privilege over it.
Long answer but in short: works with privileged container if path-issue if fixed.
//cc @stgraber


(Hagen Kuehn) #3

From your response I gather there is a known path issue when using Snap to install LXD.

Actually, when writing this post initially, I have only run the container with security.nesting=true option but it was not privileged. I have now tried the same with the additional security.privileged=true option and can confirm it’s not working.

Is there a place (bug report) where I can track the progress?

I may resort of install LXD without the use of Snap in the meantime…


(Christian Brauner) #4

If docker is run in a snap or on the host. If so you might just need to do /var/run/docker.sock.


(Hagen Kuehn) #5

Yes, I had tried with both. With source=/var/run/docker.sock and source=/var/lib/snapd/hostfs/var/run/docker.sock.

I also tried with Docker installed with Apt as well as Snap. The outcome remains the same…

$ lxc launch myimage mycontainer -c security.nesting=true -c security.privileged=true
$ lxc stop mycontainer
$ lxc config device add mycontainer dockerbladevice disk path=/var/run/docker.sock source=/var/run/docker.sock
$ lxc start mycontainer
Error: Missing source ‘/var/run/docker.sock’ for disk 'dockerbladevice’
Try lxc info --show-log mycontainer for more info


(Stéphane Graber) #6

That’s an interesting edge case I suspect. The problem may be that /run is typically a tmpfs so we may not be seeing the right copy of that path from within the LXD snap.

I’ll do some testing with the LXD and Docker snaps to see what may be going on here.


(Stéphane Graber) #7

Ok, so I think you’re going to have better luck if you use /run/docker.sock instead of /var/run/docker.sock, the reason for that is that /var/run is a symlink to /run but resolving that symlink may be made tricky due to all the namespaces used by lxd and snapd.


(Stéphane Graber) #8
root@vm10:~# lxc launch ubuntu:16.04 c1
Creating c1
Starting c1
root@vm10:~# ls -lh /run/docker.sock 
srw-rw---- 1 root root 0 Apr 27 21:23 /run/docker.sock
root@vm10:~# chmod 666 /run/docker.sock 
root@vm10:~# lxc config device add c1 docker disk source=/run/docker.sock path=/mnt/docker.sock
Device docker added to c1
root@vm10:~# lxc exec c1 -- curl --unix-socket /mnt/docker.sock docker/containers/json
[]

Note that I’m not mounting the socket onto /run/docker.sock in the container, the reason for that is that the container itself mounts its /run so any device passed into /run will be covered by a tmpfs and end up invisible.

You can then have an init script in the container which effectively does:

touch /run/docker.sock
mount --move /mnt/docker.sock /run/docker.sock
rm /mnt/docker.sock

Which will put it in the expected spot (assuming there’s no way to tell Docker where to look).
Also note that I had to change the permissions on the socket, without that, an unprivileged container won’t be able to connect.

Lastly, note that Docker will need to be started before the container for the socket to exist and that the container will have to be restarted if Docker itself gets restarted. That’s because sockets get replaced rather than reused, meaning you need a new bind-mount every time Docker is restarted.


(Christian Brauner) #9

Interesting, you did get away with ACLs? That’s kind-of unexpected to be honest. I would’ve thought that owning user namespace would be the restriction.


(Hagen Kuehn) #10

That’s great and very useful. I am away for a week but will check this out right on my return.


(Hagen Kuehn) #11

Thanks, this works perfectly!