Issues with Docker

I’ve been having some issues with Docker running inside an LXD container.

I have a production host on 4.0 from source release and a dev host on 4.1 from Snap.

On the production host I get the following issues pulling images.

root@tachikoma:~# docker pull gitlab/gitlab-ce:latest
latest: Pulling from gitlab/gitlab-ce
e92ed755c008: Pull complete
b9fd7cb1ff8f: Extracting [==================================================>]     526B/526B
ee690f2d57a1: Download complete
53e3366ec435: Download complete
76a5b707d795: Download complete
5fe252e809eb: Download complete
5fbc93a76f0d: Download complete
ccf0070fad52: Download complete
376b78f93146: Download complete
3fca551d5e4b: Download complete
failed to register layer: Error processing tar file(exit status 1): failed to mknodChar0UserNS("/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_InRelease"): failed to unlink /var/lib/apt/lists/mc0o265406352/m/d: remove /var/lib/apt/lists/mc0o265406352/m/d: operation not permitted

The same operation on my dev node works perfectly fine.

root@maran-dev:~# docker pull gitlab/gitlab-ce:latest
latest: Pulling from gitlab/gitlab-ce
e92ed755c008: Pull complete
b9fd7cb1ff8f: Pull complete
ee690f2d57a1: Pull complete
53e3366ec435: Pull complete
76a5b707d795: Pull complete
5fe252e809eb: Pull complete
5fbc93a76f0d: Pull complete
ccf0070fad52: Pull complete
376b78f93146: Pull complete
3fca551d5e4b: Pull complete
Digest: sha256:0e8e8c3842fc003911a72851a06c0aee6181ce92ac1b580d2034b9cc5741bacb
Status: Downloaded newer image for gitlab/gitlab-ce:latest

I’m guessing it’s some setting I forgot about on the prod server. Both have security.nesting set to true.

Anybody has a clue what’s going on?

You may need to turn on syscall interception, specifically the mknod and setxattr part.

Any chance you could ELI5 that for me? :slight_smile:

lxc config set NAME security.syscalls.intercept.mknod true
lxc config set NAME security.syscalls.intercept.setxattr true
lxc restart NAME

On dev enabling these settings works fine. On prod I can’t start the container anymore because of Common start logic: System doesn't support syscall interception.

I span up a few more dev VPSs to play around on to see if I can lock it down.

First I replicated my prod environment ensuring lxd and lxc were build from the 4.0 tarballs and making sure the kernel was the same. On this VPS docker would also not pull (certain) images.

I upgraded to LXD 4.1 and LXC 4.0.2 from source but Docker images would still not work.

Next I installed LXD using Snap which I had done in my working dev environment. With this version Docker started working just fine in the container.

Could you have any idea what I am doing wrong with my source builds that prevents this from working?

You might see differences here:

System call interception needs a lot of right bits:

  • Current master (unreleased) version of libseccomp
  • liblxc 4.0.0 or higher built with seccomp
  • Recent go-lxc
  • LXD 4.0 or higher
  • A 5.3 kernel or higher

The snap has all the right bits minus the kernel which we can’t control.

1 Like

I want to ask something. Is there a way to use kubernetes to orchestrate linux containers ?
From a quick search, i found that you can install kubernetes inside linux containers , but you can’t manage linux containers with kubernetes. Am i wrong ?

Ive never used it, but yes I think you can manage linux containers with Kubernetes

Thanks a lot. I will try to test it.

Since everything seems up-to-date except libseccomp I think that’s the culprit.

I tried my best to compile lxc against libseccomp from master but it keeps using the server wide version. Does anybody have any tips how to use the specific libseccomp I need?

I’ve tried LDFLAGS="-L/root/libseccomp/src/.libs/" ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var

Perhaps I’m approaching this the wrong way. The main issue I’m trying to solve is that I need a newer Ceph than is bundled in the Snap. Can I use a Ceph outside of the Snap somehow?