Increase SOMAXCONN in unprivileged containers?


(Evgeniy K.) #1

Hello, all.

Is there a way to increase net.core.somaxconn in unprivileged containers?

Based on my short research this key was “namespaced” and very long time ago. So, it should be available for tuning. And it could be easily confirmed with “ip netns …” (change somaxconn, create new ns and change in it value as you want).
But when you’re using unprivileged container you also have separate user namespace. Using of userns breaks it for LXD-managed containers and even for Docker (w/enabled userns). And things will get even worse, when you read “man listen”:

If the backlog argument is greater than the value in /proc/sys/net/core/somaxconn, then it is silently truncated to that value; the default value in this file is 128. In kernels before
2.4.25, this limit was a hard coded value, SOMAXCONN, with the value 128.

So, you end up with socket and 128 as default value for sockets backlog. People with Kubernetes found a workaround with privileged init-containers. For example this article - http://bogdan-albei.blogspot.com/2017/09/kernel-tuning-in-kubernetes.html. It looks like a huge headache, but it fixes the problem.

Not sure, that I was digging at right direction, but ended up with these:


It looks like it was “broken” very long time ago and based commit’s commentary it was made intentionally (i.e. it’s not a bug). So, question still remains - is there a way to achieve security and performance? I.e. tune somaxconn w/out creating privileged container?


(Stéphane Graber) #2

It may be possible for LXD itself to modify such sysctls as real root in the container, the key would then need to be exposed through a limits. config key and code be added for LXD to perform that change.


(Stéphane Graber) #3

@brauner thoughts? Feels like we could have a LXD fork command which attaches to the netns, unshares a mntns, mounts proc and messes with sysctls, that should work for network sysctls.


(Evgeniy K.) #4

It would be great to have this. For example, Docker already has feature in their own way:

docker run --sysctl net.ipv4.ip_forward=1 someimage

Took it from here:


(Evgeniy K.) #6

To clarify - example above doesn’t work when you have enabled “userns-remap” option in dockerd (it enables the same behavior as LXD does with unprivileged containers, i.e. userns plus subuid/subgid). So, example from my previous comment just as a demonstration of Docker’s CLI option and corresponding section in “compose” config. Not sure that it could help at any way, but I hope that it would.


(Christian Brauner) #7

You mean global sysctls for the whole system? That sounds like a bad idea imho.


(Evgeniy K.) #8

Probably @stgraber meant “namespaced” sysctl keys. Can’t quickly find full list, but here is a small example between default and custom network namespace in meaning of sysctl keys.

Default network namespace:

root@hostname:~# sysctl net 2>/dev/null | wc -l
719

root@hostname:~# sysctl net.core.wmem_default
net.core.wmem_default = 212992

root@hostname:~# sysctl net.core.somaxconn
net.core.somaxconn = 128
root@hostname:~# sysctl net.core.somaxconn=129
net.core.somaxconn = 129
root@hostname:~# sysctl net.core.somaxconn
net.core.somaxconn = 129

Custom namespace:

root@hostname:~# ip netns add test
root@hostname:~# ip netns exec test bash
root@hostname:~# sysctl net 2>/dev/null | wc -l
402
root@hostname:~# sysctl net.core.wmem_default
sysctl: cannot stat /proc/sys/net/core/wmem_default: No such file or directory

root@hostname:~# sysctl net.core.somaxconn
net.core.somaxconn = 128
root@hostname:~# sysctl net.core.somaxconn=130
net.core.somaxconn = 130
root@hostname:~# sysctl net.core.somaxconn
net.core.somaxconn = 130

And default again:

root@hostname:~# sysctl net.core.somaxconn
net.core.somaxconn = 129

I hope the difference is clearly visible (inside of custom namespace you can see and tune only “namespaced” keys). And changes doesn’t affect the host system (correct me if I mistaken here).

And repeating my initial question: is there a way to tune net.core.somaxconn inside of unprivileged LXC container (in custom network namespace)?


(Christian Brauner) #9

Every sysctl that is properly namespaced can be accessed in LXC through the

lxc.sysctl.{kernel parameter name}

key. So in your case you’d need to set:

lxc.sysctl.net.core.somaxconn = <value>

The question is whether somaxconn is namespaced or not. I don’t see it show up in an unprivileged container under /proc/sys/net/core/. So I doubt it is. If it is namespaced but people forgot to also namespace the sysctl then this is a kernel bug and should be fixed there.


(Evgeniy K.) #10

The question is whether somaxconn is namespaced or not. I don’t see it show up in an unprivileged container under /proc/sys/net/core/ . So I doubt it is. If it is namespaced but people forgot to also namespace the sysctl then this is a kernel bug and should be fixed there.

As I show above, you can see it in separate network namespace (aslo as non-root user). But the main problem in using “users namespace”. As soon as you in unprivileged LXC-container or equal Docker-container w/enabled userns-remap - you can’t see this key. And as I wrote before, I tried to find out where or how it in kernel, but got lost deep in the sources.

This is exactly the reason why I initially asked “Is there a way …?”. Because I’m not so familiar with container’s bootstrap process (how and when namespaces/mounts/interfaces are created).

And right now we have only two options - to choose security (i.e. unprivileged) or performance (privileged+tuning). But after reading your article about risks with privileged containers (or even super-privileged in case of Docker) our doubts are even stronger. In the same time we can’t sacrifice performance and end up with containerized applications, which can’t even handle medium load. It’s a dead end as I see it, to be honest. To be clear - I’m not asking anyone to solve our problems. I’m just asking for advise about possible workaround in such controversial situation.

Thanks.


(Christian Brauner) #11

So just to clear things up a little. We are dealing with two scenarios:

  1. non-initial network namespace owned by the initial user namespace
  2. non-initial network namespace owned by a non-initial user namespace

For 1. you can change somaxconn because you need to be privileged enough in the initial user namespace. For 2. you can’t change somaxconn because that would mean giving a non-initial user namespace the ability to affect a system-wide setting. This is a big no-no. So the real solution here to me seems to bump the somaxconn setting globally before starting any container to a large enough value that you don’t run into this issue.


(Evgeniy K.) #12

Yep, we also tried as you described and it didn’t work. I.e. even if we bumped up value before creating container. We get “hardcoded” 128 all the time. Here is a small example:

root@hostname:~# sysctl net.core.somaxconn
net.core.somaxconn = 130

root@hostname:~# lxc init ubuntu:16.04 test
Creating test

root@hostname:~# lxc config set test security.privileged true
root@hostname:~# lxc config show test | grep privileged
  security.privileged: "true"

root@hostname:~# lxc start test
root@hostname:~# lxc exec test bash
root@test:~# sysctl net.core.somaxconn
net.core.somaxconn = 128

root@hostname:~# sysctl net.core.somaxconn
net.core.somaxconn = 130

So, it’s bump to 130 => create container => enter => see 128 instead of 130. And probably because of this:


PS: I made container privileged just to show “default” value in container’s namespace. As we discussed above, you can’t even see this key in unprivileged.


(Christian Brauner) #13

Hm, you can try:

lxc config set <container-name> raw.lxc 'lxc.sysctl.net.core.somaxconn = 1000'

and restart a privileged container.


(Evgeniy K.) #14

And as expected it worked:

root@hostname:~# lxc init ubuntu:16.04 test
Creating test
root@hostname:~# lxc config set test security.privileged true
root@hostname:~# lxc config set test raw.lxc 'lxc.sysctl.net.core.somaxconn = 1000'
root@hostname:~# lxc start test
root@hostname:~# lxc exec test bash
root@test:~# sysctl net.core.somaxconn
net.core.somaxconn = 1000

But it’s not a problem. Anyone can achieve same thing just by setting same key via /etc/sysctl.conf at any distro. The initial problem still remains - is it possible to do the same for unprivileged? I still can’t understand the difference between unprivileged users in default namespace and same users in custom namespace. Here is what I’m talking about:

root@hostname:~# ip netns add test
root@hostname:~# ip netns exec test bash
root@hostname:~# sysctl net.core.somaxconn
net.core.somaxconn = 128
root@hostname:~# sudo -u nobody /bin/bash
nobody@hostname:~$ id
uid=65534(nobody) gid=65534(nogroup) groups=65534(nogroup)
nobody@hostname:~$ sysctl net.core.somaxconn
net.core.somaxconn = 128
nobody@hostname:~$ sysctl net.core.somaxconn=129
sysctl: permission denied on key 'net.core.somaxconn'

Yes, we can’t change the value as unprivileged user. But in the same time we still can even see it (even as non-root user). So, how it’s different from lxd as unprivileged user and his processes?

root@hostname:~# grep lxd /etc/subuid
lxd:165536:65536
root@hostname:~# ps auxf | grep -A1 '[t]est'
root      5799  0.0  0.0 205144  7160 ?        Ss   02:56   0:00 [lxc monitor] /var/lib/lxd/containers test
165536    5815  0.5  0.0  37472  5580 ?        Ss   02:56   0:00  \_ /sbin/init

When we’re in container, we’re in different user and network namespaces (plus more). But why we can’t even see the sysctl key, which awlays exists in any additional network namespace by default?


(Evgeniy K.) #15

Hmm … Looks like here is a difference:

root@hostname:~# ls -lA /proc/$$/ns
total 0
lrwxrwxrwx 1 root root 0 Apr  5 03:08 cgroup -> cgroup:[4026531835]
lrwxrwxrwx 1 root root 0 Apr  5 03:08 ipc -> ipc:[4026531839]
lrwxrwxrwx 1 root root 0 Apr  5 03:08 mnt -> mnt:[4026531840]
lrwxrwxrwx 1 root root 0 Apr  5 03:08 net -> net:[4026532009]
lrwxrwxrwx 1 root root 0 Apr  5 03:08 pid -> pid:[4026531836]
lrwxrwxrwx 1 root root 0 Apr  5 03:08 pid_for_children -> pid:[4026531836]
lrwxrwxrwx 1 root root 0 Apr  5 03:08 user -> user:[4026531837]
lrwxrwxrwx 1 root root 0 Apr  5 03:08 uts -> uts:[4026531838]

root@hostname:~# ip netns exec test bash
root@hostname:~# ls -lA /proc/$$/ns
total 0
lrwxrwxrwx 1 root root 0 Apr  5 03:08 cgroup -> cgroup:[4026531835]
lrwxrwxrwx 1 root root 0 Apr  5 03:08 ipc -> ipc:[4026531839]
lrwxrwxrwx 1 root root 0 Apr  5 03:08 mnt -> mnt:[4026533082]
lrwxrwxrwx 1 root root 0 Apr  5 03:08 net -> net:[4026532889]
lrwxrwxrwx 1 root root 0 Apr  5 03:08 pid -> pid:[4026531836]
lrwxrwxrwx 1 root root 0 Apr  5 03:08 pid_for_children -> pid:[4026531836]
lrwxrwxrwx 1 root root 0 Apr  5 03:08 user -> user:[4026531837]
lrwxrwxrwx 1 root root 0 Apr  5 03:08 uts -> uts:[4026531838]

root@hostname:~# ps auxf | grep -A1 '[t]est'
root      5799  0.0  0.0 205144  7160 ?        Ss   02:56   0:00 [lxc monitor] /var/lib/lxd/containers test
165536    5815  0.0  0.0  37472  5580 ?        Ss   02:56   0:00  \_ /sbin/init
root@hostname:~# ls -lA /proc/5815/ns
total 0
lrwxrwxrwx 1 165536 165536 0 Apr  5 02:56 cgroup -> cgroup:[4026533078]
lrwxrwxrwx 1 165536 165536 0 Apr  5 02:56 ipc -> ipc:[4026532996]
lrwxrwxrwx 1 165536 165536 0 Apr  5 02:56 mnt -> mnt:[4026532994]
lrwxrwxrwx 1 165536 165536 0 Apr  5 02:56 net -> net:[4026532999]
lrwxrwxrwx 1 165536 165536 0 Apr  5 02:56 pid -> pid:[4026532997]
lrwxrwxrwx 1 165536 165536 0 Apr  5 03:05 pid_for_children -> pid:[4026532997]
lrwxrwxrwx 1 165536 165536 0 Apr  5 02:56 user -> user:[4026532993]
lrwxrwxrwx 1 165536 165536 0 Apr  5 02:56 uts -> uts:[4026532995]

Ordered as “default”, “only net namespace changed” and “all namespaces changed”. And it looks like as soon as you changed your net+user namespaces you instantly lost access to /proc/sys/net/core (it’s empty, but viewable).


(Christian Brauner) #16

That’s what I tried to explain before. Probably not clear enough, which is my bad. The kernel has a concept of ownership for various namespaces including network namespaces. When you ask the question “Am I allowed to perform an operation on this network namespace?”. The kernel will check who owns the network namespace in question. In the case of a privileged container the answer will be that the owner is the initial user namespace which is where global root lives. In the case of an unprivileged container the answer is that the owner is a non-initial user namespace which is not nearly as privileged as the initial user namespace. So the kernel will not allow you to even get near settings that are controlled by the initial user namespace.


(Evgeniy K.) #17

Probably I’m missing the difference between “why we can’t see net.core.somaxconn”, but we still can see and even change many other sysctl keys:

root@test:~# sysctl -e net 2>/dev/null | head -n 10
net.ipv4.conf.all.accept_local = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_notify = 0
net.ipv4.conf.all.bootp_relay = 0
net.ipv4.conf.all.disable_policy = 0

root@test:~# sysctl net.ipv4.conf.all.accept_local=1
net.ipv4.conf.all.accept_local = 1

Even as unprivileged user, i.e. fake-root. So, how somaxconn is different? It’s a namespaced key and even if it changed, it won’t affect initial network namespace (i.e. host system).

PS: Maybe it’s a wrong place to ask such questions, but I’m still trying to wrap my head around all this.


(Christian Brauner) #18

I don’t know what the exact reasoning behind this is but the kernel definitely doesn’t let you access any net/core sysctls at all.


#19

well you found it already it’s just straight hardcoded in from
net/core/net_namespace.c:

static int __net_init net_defaults_init_net(struct net *net)
{
        net->core.sysctl_somaxconn = SOMAXCONN;
        return 0;
}

That’s the default value that is used and after that the sysctl comes and changes it. but not for containers, because of this - I tried to remove it and afterwards the root user of an unprivileged container could change the value for the container (and only the container - the host value is unchanged and other containers are not affected too)
In fact everything is almost perfect like this, the only part that could be better is an option to leave the ‘file’ /proc/sys/net/core/somaxconn owned by the ‘real’ root so it could be setup by the lxd manager (with the raw.lxc key) and only used by the container software. I tried that and it definitely works.

It seems definitely wrong to hide the key. It should be owned by the global root instead.

As of your problem, my guess is that if you need a way, you can always compile your own kernel and setup the default value SOMAXCONN higher. It will not really hurt standard software, just eat resources as the default queues may be higher (if the application software don’t set the value itself to a saner value). Maintenance headeache yes, but it beats a non working system.