Device config: listener on bridge being started inside container fails

LXD
1

In common I provide access to mailservers on a lxd host via container specific proxy configurations that listen on 127.0.0.1:25 inside the container and connect to 127.0.0.1:25 on the host.

With running docker inside a lxc container the docker containers know their docker0 at 172.17.0.1 bridge for networking, so I added the following device config to enable sending emails from inside docker containers:

lxc config device add mycontainer mailserver-docker proxy listen=tcp:172.17.0.1:25 connect=tcp:127.0.0.1:25 bind=container

This is a working setup. But things break when the lxc container needs to be (re)started: lxd tries to setup the device ‘mailserver-docker’ and cannot listen to 172.17.0.1:25 because the docker bridge is started later on in the staring process (Error: Error occurred when starting proxy device: Error: Failed to listen on 172.17.0.1:25: listen tcp 172.17.0.1:25: bind: cannot assign requested address).

Now I think about different approaches:

  1. use some docker-specific mechanism to bind port 25 to the lxc-specific port 25 that connects to port 25 on the host? Hint: I didn’t find anything appropriate.
  2. checkout if lxd accepts scripts to be executed on shutdown (‘device remove’) and startup (repeatedly try to ‘device add’ 
 not very nice)
  3. ask for an extension of device add proxy by the LXD team: if listening fails the container could be started with a warning only and the device could be attached later on with some retries.

Any (further) suggestions?

For background information:

# lxc ls -c ns4 mycontainer
+-------------------+---------+---------------------------------+
|       NAME        |  STATE  |              IPV4               |
+-------------------+---------+---------------------------------+
| mycontainer       | RUNNING | 192.168.96.1 (br-67879c888ac8)  |
|                   |         | 192.168.64.1 (br-467e39ed58ee)  |
|                   |         | 192.168.48.1 (br-0e0d44af9da9)  |
|                   |         | 192.168.32.1 (br-6b66559b7ca7)  |
|                   |         | 192.168.208.1 (br-78146addde19) |
|                   |         | 192.168.144.1 (br-574905d87763) |
|                   |         | 192.168.112.1 (br-7017a17107e3) |
|                   |         | 192.168.0.1 (br-187bf916c8d8)   |
|                   |         | 172.31.0.1 (br-48e056798521)    |
|                   |         | 172.30.0.1 (br-ec9b4eb276b1)    |
|                   |         | 172.29.0.1 (br-d7d7e96fcbc3)    |
|                   |         | 172.28.0.1 (br-ff3430e2408f)    |
|                   |         | 172.25.0.1 (br-b754a3b64228)    |
|                   |         | 172.24.0.1 (br-d4fbc8315c5d)    |
|                   |         | 172.23.0.1 (br-ec45feafd8ec)    |
|                   |         | 172.20.0.1 (br-a63dcaf4beac)    |
|                   |         | 172.19.0.1 (br-668398d3183b)    |
|                   |         | 172.18.0.1 (br-b0f213c37afb)    |
|                   |         | 172.17.0.1 (docker0)            |
|                   |         | 10.187.210.165 (eth0)           |
+-------------------+---------+---------------------------------+

# lxc config device list mycontainer
mailserver
mailserver-docker

# lxc stop mycontainer

# lxc start mycontainer
Error: Error occurred when starting proxy device: Error: Failed to listen on 172.17.0.1:25: listen tcp 172.17.0.1:25: bind: cannot assign requested address
Try `lxc info --show-log mycontainer` for more info

My Workaround:

# lxc config device remove mycontainer mailserver-docker
Device mailserver-docker removed from mycontainer
# lxc start mycontainer
# lxc config device add mycontainer mailserver-docker proxy listen=tcp:172.17.0.1:25 connect=tcp:127.0.0.1:25 bind=container
Device mailserver-docker added to mycontainer
1 Like
2

If you set the sysctl net.ipv4.ip_nonlocal_bind=1 inside the container then the proxy device should be able to listen on a non-existent IP, and then once the IP comes up the socket should work.

You can enable this manually inside the container using:

sysctl -w net.ipv4.ip_nonlocal_bind=1

Or you can use LXD instance setting raw.lxc so it applies on startup:

lxc stop <instance>
lxc config set <instance> raw.lxc="lxc.sysctl.net.ipv4.ip_nonlocal_bind=1"
lxc start <instance>

Then check its applied:

lxc exec <instance> -- sysctl -a | grep ip_nonlocal_bind
net.ipv4.ip_nonlocal_bind = 1
net.ipv6.ip_nonlocal_bind = 0
1 Like
3

@stgraber out of interest, is there a reason why LXD doesn’t provide the ability to set sysctls inside the container without using raw.lxc? Could be similar to environment.* config keys?

4

Thanks Tom, works like a charm!

1 Like
5

I wouldn’t be opposed to adding a linux.sysctl.* type key, similar to limits.kernel.* which sets the kernel limits.

1 Like
6

@tomp can you add something to the ideas board for it?

7

Yep will do :slight_smile: