Error with docker inside lxc container

LXD
1

Hello folks!

I’m facing with a weird problem with my server, I’m using LXC/LXD for a while with a good results overall, but since yesterday (no updates were installed) I’m having a lot of troubles when triying to create or start a docker instance.

If I run (inside a lxc container) this
$ docker run hello-world

docker: Error response from daemon: oci runtime error: could not create session key: disk quota exceeded.

this is the current version of docker (insude LXC container)

Client:
Version: 1.12.3
API version: 1.24
Go version: go1.6.2
Git commit: 6b644ec
Built: Mon, 19 Dec 2016 09:20:48 +1300
OS/Arch: linux/amd64

Server:
Version: 1.12.3
API version: 1.24
Go version: go1.6.2
Git commit: 6b644ec
Built: Mon, 19 Dec 2016 09:20:48 +1300
OS/Arch: linux/amd64

this is the version of LXC

root@xxxx:~# lxc version

2.0.9

I’ve checked almost everything (like free space, inodes, etc and everything looks fine) I have ext4 as filesystem (so no disk quota at all)

$ df -i

Filesystem Inodes IUsed IFree IUse% Mounted on
udev 8229243 499 8228744 1% /dev
tmpfs 8234072 1970 8232102 1% /run
/dev/md1 27992064 6655709 21336355 24% /
tmpfs 8234072 1 8234071 1% /dev/shm
tmpfs 8234072 4 8234068 1% /run/lock
tmpfs 8234072 16 8234056 1% /sys/fs/cgroup
/dev/md2 1281120 13 1281107 1% /home
tmpfs 8234072 4 8234068 1% /run/user/0

Any thoughts?

Outside LXC container docker works as expected (tested using hello-world docker)

If you need further information please let me know.

Thank you.

2

Kinda sounds like Docker may be attempting to use the kernel keyring?
That’d certainly be a new behavior from them…

Unfortunately you version of LXD doesn’t support syscall blacklisting so it’s not particularly easy to test/workaround in your case…

Where did you get that version of Docker?

3

Hello!

That docker version is from docker.io repo

I've found the solution; increasing /proc/sys/kernel/keys/maxkeys from 200 to a higher value (50000) fixed the problem, I think that it was because I have almost 30 lxc containers running and all of them are unprivileged.

Many thanks for your help!

4

Ah, that’s good to know.

5

Can confirm as of Dec 2019 that this works - do sudo echo "5000" > /proc/sys/kernel/keys/maxkeys on your LXD host.

6

The command should rather be something like

echo 5000 | sudo tee /proc/sys/kernel/keys/maxkeys
7

still prolem as of 2022 , lxd should come with a profile setup for dockers.

8

This is covered in Production setup - LXD documentation

9

Thanks , here i made a copy-pastable version for new comers.

fs.aio-max-nr=524288
fs.inotify.max_queued_events=1048576
fs.inotify.max_user_instances=1048576
fs.inotify.max_user_watches=1048576
kernel.dmesg_restrict=1
kernel.keys.maxbytes=2000000
kernel.keys.maxkeys=20000
net.ipv4.neigh.default.gc_thresh3=8192
net.ipv6.neigh.default.gc_thresh3=8192
vm.max_map_count=262144

Another problem is Ulimit :
add to /etc/security/limits.conf

    * soft core unlimited
    * hard core unlimited
    * soft data unlimited
    * hard data unlimited
    * soft fsize unlimited
    * hard fsize unlimited
    * soft memlock unlimited
    * hard memlock unlimited
    * soft nofile 1048576
    * hard nofile 1048576
    * soft rss unlimited
    * hard rss unlimited
    * soft stack unlimited
    * hard stack unlimited
    * soft cpu unlimited
    * hard cpu unlimited
    * soft nproc unlimited
    * hard nproc unlimited
    * soft as unlimited
    * hard as unlimited
    * soft maxlogins unlimited
    * hard maxlogins unlimited
    * soft maxsyslogins unlimited
    * hard maxsyslogins unlimited
    * soft locks unlimited
    * hard locks unlimited
    * soft sigpending unlimited
    * hard sigpending unlimited
    * soft msgqueue unlimited
    * hard msgqueue unlimited
10

This is now available at Production setup - LXD documentation