We are using LXD, among others, to have a local development environment that can be shared between developers. Without a fuse, we can run Docker completely isolated within the LXD/ LXC container. That’s great! Those LXD/ LXC containers are short lived and used for only one development task.
In order to reduce network load and traffic charges, we would like to “run” Docker within the LXD/ LXC container and actually use the Docker instance running on the host machine (typically its run on a developer’s laptop that runs Debian/ Ubuntu). This way the Docker images (or slices thereof) don’t need to be downloaded again since executing Docker commands within the LXD/ LXC container are actually using the host’s Docker. Basically, we want the equivalent of docker run -v /var/run/docker.sock:/var/run/docker.sock ... in LXD/ LXC.
We tried the below without success:
Created LXD/ LXC container (still stopped)
Added device with either one of the below commands
lxc config device add container-name dockerbladevice disk path=/var/run/docker.sock
source=/var/lib/snapd/hostfs/var/run/docker.sock
lxc config device add container-name dockerbladevice disk path=/var/run/docker.sock
source=/var/run/docker.sock
Results in error
$ lxc start container-name
Error: Missing source '/var/lib/snapd/hostfs/var/run/docker.sock' for disk 'dockerbladevice'
Try `lxc info --show-log container-name` for more info
Info output
$ lxc info --show-log container-name
Name: container-name
Remote: unix://
Architecture: x86_64
Created: 2018/04/27 10:26 UTC
Status: Stopped
Type: persistent
Profiles: default
Log:
lxc 20180427102823.671 WARN lxc_conf - conf.c:lxc_map_ids:2831 - newuidmap binary is missing
lxc 20180427102823.671 WARN lxc_conf - conf.c:lxc_map_ids:2837 - newgidmap binary is missing
lxc 20180427102823.685 WARN lxc_conf - conf.c:lxc_map_ids:2831 - newuidmap binary is missing
lxc 20180427102823.686 WARN lxc_conf - conf.c:lxc_map_ids:2837 - newgidmap binary is missing
lxc 20180427102823.688 WARN lxc_cgfsng - cgroups/cgfsng.c:chowmod:1881 - No such file or directory -
Failed to chown(/sys/fs/cgroup/unified//lxc/container-name/cgroup.threads, 1000000000, 0)
lxc 20180427103203.468 WARN lxc_conf - conf.c:lxc_map_ids:2831 - newuidmap binary is missing
lxc 20180427103203.468 WARN lxc_conf - conf.c:lxc_map_ids:2837 - newgidmap binary is missing
This looks like a path-based issue with snapd. That’s something @stgraber can surely help with. Because - at least with a privileged container - this works fine for me. I don’t expect it to work for an unprivileged container because sockets have a owning user namespace which in this case will be the initial user namespace but an unprivileged container will have a separate user namespace. So the kernel will check your permissions against the owning user namespace of the socket (which is the initial user namespace) but your container won’t hold any privilege over it.
Long answer but in short: works with privileged container if path-issue if fixed.
//cc @stgraber
From your response I gather there is a known path issue when using Snap to install LXD.
Actually, when writing this post initially, I have only run the container with security.nesting=true option but it was not privileged. I have now tried the same with the additional security.privileged=true option and can confirm it’s not working.
Is there a place (bug report) where I can track the progress?
I may resort of install LXD without the use of Snap in the meantime…
That’s an interesting edge case I suspect. The problem may be that /run is typically a tmpfs so we may not be seeing the right copy of that path from within the LXD snap.
I’ll do some testing with the LXD and Docker snaps to see what may be going on here.
Ok, so I think you’re going to have better luck if you use /run/docker.sock instead of /var/run/docker.sock, the reason for that is that /var/run is a symlink to /run but resolving that symlink may be made tricky due to all the namespaces used by lxd and snapd.
Note that I’m not mounting the socket onto /run/docker.sock in the container, the reason for that is that the container itself mounts its /run so any device passed into /run will be covered by a tmpfs and end up invisible.
You can then have an init script in the container which effectively does:
touch /run/docker.sock
mount --move /mnt/docker.sock /run/docker.sock
rm /mnt/docker.sock
Which will put it in the expected spot (assuming there’s no way to tell Docker where to look).
Also note that I had to change the permissions on the socket, without that, an unprivileged container won’t be able to connect.
Lastly, note that Docker will need to be started before the container for the socket to exist and that the container will have to be restarted if Docker itself gets restarted. That’s because sockets get replaced rather than reused, meaning you need a new bind-mount every time Docker is restarted.
Interesting, you did get away with ACLs? That’s kind-of unexpected to be honest. I would’ve thought that owning user namespace would be the restriction.
This no longer works in 2023. Is there a new solution to this problem? I’d really like to see the hosts docker environment in the container. If no longer possible, I guess I’ll learn to live with it…
Attempting to run:
mount --move /mnt/docker.sock /run/docker.sock
results in:
mount: /run/docker.sock: bad option; moving a mount residing under a shared mount is unsupported.
Ubuntu 20.04.4 LTS on host and 20.04.5 LTS in container.
Hmm - I wonder if I need to match the host and container OS’…
I tried something and it seems it just might work:
mount --bind /mnt/docker.sock /run/docker.sock
It seems to give me the view I’m after, but a container I built after doing this failed to run. It could be just a coincidence that there’s a bug in my code, but it’d be awesome to see if @stgraber believes the workaround I stumbled upon is sufficient.
UPDATE: Fixed the bug causing container not to run. So, this work around appears solid. I say Ship It!