Run docker on lxd container

LXD
,
1

I have searched quite a lot about this but I’m not satisfied with the results as per most were old posts. My LXD is running on a ZFS pool and I would like to run docker on a container. Initially I had no problems to setup docker I simply ran:

lxc launch images:debian/10 v1 -c security.privileged=true -c security.nesting=true
lxc exec v1 bash
curl -fsSL get.docker.com | sh

The docker storage drive is set to vfs and I would like to change this to overlay. I am passing the linux kernel to the container :

lxc config set v1 linux.kernel_modules overlay

and inside the container I updated /etc/docker/daemon.json:

{
  "storage-driver": "overlay2"
}

When I start the docker service again, the service does not start.

Log from journalctl -f

Jul 13 20:15:42 v1 systemd[1]: Starting Docker Application Container Engine...
Jul 13 20:15:42 v1 dockerd[696]: time="2021-07-13T18:15:42.283024071Z" level=info msg="Starting up"
Jul 13 20:15:42 v1 dockerd[696]: time="2021-07-13T18:15:42.288495013Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Jul 13 20:15:42 v1 dockerd[696]: time="2021-07-13T18:15:42.288670863Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jul 13 20:15:42 v1 dockerd[696]: time="2021-07-13T18:15:42.288746584Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Jul 13 20:15:42 v1 dockerd[696]: time="2021-07-13T18:15:42.288815305Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jul 13 20:15:42 v1 dockerd[696]: time="2021-07-13T18:15:42.294450060Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Jul 13 20:15:42 v1 dockerd[696]: time="2021-07-13T18:15:42.294814092Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jul 13 20:15:42 v1 dockerd[696]: time="2021-07-13T18:15:42.294933997Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Jul 13 20:15:42 v1 dockerd[696]: time="2021-07-13T18:15:42.294980552Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jul 13 20:15:42 v1 dockerd[696]: time="2021-07-13T18:15:42.310135470Z" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay2
Jul 13 20:15:42 v1 dockerd[696]: failed to start daemon: error initializing graphdriver: driver not supported
Jul 13 20:15:42 v1 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jul 13 20:15:42 v1 systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 13 20:15:42 v1 systemd[1]: Failed to start Docker Application Container Engine.
Jul 13 20:15:44 v1 systemd[1]: docker.service: Service RestartSec=2s expired, scheduling restart.
Jul 13 20:15:44 v1 systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Jul 13 20:15:44 v1 systemd[1]: Stopped Docker Application Container Engine.
Jul 13 20:15:44 v1 systemd[1]: docker.service: Start request repeated too quickly.
Jul 13 20:15:44 v1 systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 13 20:15:44 v1 systemd[1]: Failed to start Docker Application Container Engine.
Jul 13 20:15:44 v1 systemd[1]: docker.socket: Failed with result 'service-start-limit-hit'.

What can I do to run docker storage driver overlay2 on an lxd container?

2

Because it’s seems unsupported : Docker with overlay driver in LXD cluster not working - #2 by stgraber

3

Hmm I see yes, thanks for that reference @ruskofd

4

It depends on the storage backend being used for the LXD container.
overlay2 is incompatible with ZFS and there’s no sign of that improving any time soon.

But if your container is stored on just about anything else, then overlay2 should work fine, especially since you’re dealing with privileged in your case.

For unprivileged, it will work too, at least so long as you run on a kernel that allows unprivileged overlayfs (Ubuntu does, not sure what other distros do these days).

Still in the unprivileged case, you’d also want to set security.syscalls.intercept.mknod=true and security.syscalls.intercept.setxattr=true to avoid some issues unpacking some Docker layers.

1 Like
5

Thanks for the info! I have switched to brtfs and had no problems this time.

2 Likes
6

Also might be worth pointing out that BTRFS backed storage also doesn’t play completely nicely with docker because docker then uses BTRFS’ native snapshotting functionality instead of overlay, which isn’t fully supported by LXD when transferring the container between hosts/pools.

See

2 Likes
7

Wow, thank you for posting this! I am about to start running CI builder agents on a server with one big BTRFS pool. Container builds with Docker (and Buildah) will be involved.

I am in a position to delete and recreate these agents on a schedule, since they’re really just stateless workers.

If I delete and recreate my containers regularly, will that be a workaround for this amplification issue?

update: ah if this is only an issue when shipping volumes from host to host, I’ll be fine.

1 Like
8

I noticed that with a 6.x kernel, with a ZFS backend, docker info shows that it uses overlay2. Is there a way to confirm that overlay2 is supported by ZFS on my docker host (other than docker info)?

And by unpacking docker layers did you mean to fix the “Failed to register docker layer…” errors?