Device config: listener on bridge being started inside container fails

In common I provide access to mailservers on a lxd host via container specific proxy configurations that listen on 127.0.0.1:25 inside the container and connect to 127.0.0.1:25 on the host.

With running docker inside a lxc container the docker containers know their docker0 at 172.17.0.1 bridge for networking, so I added the following device config to enable sending emails from inside docker containers:

lxc config device add mycontainer mailserver-docker proxy listen=tcp:172.17.0.1:25 connect=tcp:127.0.0.1:25 bind=container

This is a working setup. But things break when the lxc container needs to be (re)started: lxd tries to setup the device ‘mailserver-docker’ and cannot listen to 172.17.0.1:25 because the docker bridge is started later on in the staring process (Error: Error occurred when starting proxy device: Error: Failed to listen on 172.17.0.1:25: listen tcp 172.17.0.1:25: bind: cannot assign requested address).

Now I think about different approaches:

  1. use some docker-specific mechanism to bind port 25 to the lxc-specific port 25 that connects to port 25 on the host? Hint: I didn’t find anything appropriate.
  2. checkout if lxd accepts scripts to be executed on shutdown (‘device remove’) and startup (repeatedly try to ‘device add’ … not very nice)
  3. ask for an extension of device add proxy by the LXD team: if listening fails the container could be started with a warning only and the device could be attached later on with some retries.

Any (further) suggestions?


For background information:

# lxc ls -c ns4 mycontainer
+-------------------+---------+---------------------------------+
|       NAME        |  STATE  |              IPV4               |
+-------------------+---------+---------------------------------+
| mycontainer       | RUNNING | 192.168.96.1 (br-67879c888ac8)  |
|                   |         | 192.168.64.1 (br-467e39ed58ee)  |
|                   |         | 192.168.48.1 (br-0e0d44af9da9)  |
|                   |         | 192.168.32.1 (br-6b66559b7ca7)  |
|                   |         | 192.168.208.1 (br-78146addde19) |
|                   |         | 192.168.144.1 (br-574905d87763) |
|                   |         | 192.168.112.1 (br-7017a17107e3) |
|                   |         | 192.168.0.1 (br-187bf916c8d8)   |
|                   |         | 172.31.0.1 (br-48e056798521)    |
|                   |         | 172.30.0.1 (br-ec9b4eb276b1)    |
|                   |         | 172.29.0.1 (br-d7d7e96fcbc3)    |
|                   |         | 172.28.0.1 (br-ff3430e2408f)    |
|                   |         | 172.25.0.1 (br-b754a3b64228)    |
|                   |         | 172.24.0.1 (br-d4fbc8315c5d)    |
|                   |         | 172.23.0.1 (br-ec45feafd8ec)    |
|                   |         | 172.20.0.1 (br-a63dcaf4beac)    |
|                   |         | 172.19.0.1 (br-668398d3183b)    |
|                   |         | 172.18.0.1 (br-b0f213c37afb)    |
|                   |         | 172.17.0.1 (docker0)            |
|                   |         | 10.187.210.165 (eth0)           |
+-------------------+---------+---------------------------------+

# lxc config device list mycontainer
mailserver
mailserver-docker

# lxc stop mycontainer

# lxc start mycontainer
Error: Error occurred when starting proxy device: Error: Failed to listen on 172.17.0.1:25: listen tcp 172.17.0.1:25: bind: cannot assign requested address
Try `lxc info --show-log mycontainer` for more info

My Workaround:

# lxc config device remove mycontainer mailserver-docker
Device mailserver-docker removed from mycontainer
# lxc start mycontainer
# lxc config device add mycontainer mailserver-docker proxy listen=tcp:172.17.0.1:25 connect=tcp:127.0.0.1:25 bind=container
Device mailserver-docker added to mycontainer

If you set the sysctl net.ipv4.ip_nonlocal_bind=1 inside the container then the proxy device should be able to listen on a non-existent IP, and then once the IP comes up the socket should work.

You can enable this manually inside the container using:

sysctl -w net.ipv4.ip_nonlocal_bind=1

Or you can use LXD instance setting raw.lxc so it applies on startup:

lxc stop <instance>
lxc config set <instance> raw.lxc="lxc.sysctl.net.ipv4.ip_nonlocal_bind=1"
lxc start <instance>

Then check its applied:

lxc exec <instance> -- sysctl -a | grep ip_nonlocal_bind
net.ipv4.ip_nonlocal_bind = 1
net.ipv6.ip_nonlocal_bind = 0
1 Like

@stgraber out of interest, is there a reason why LXD doesn’t provide the ability to set sysctls inside the container without using raw.lxc? Could be similar to environment.* config keys?

Thanks Tom, works like a charm!

1 Like

I wouldn’t be opposed to adding a linux.sysctl.* type key, similar to limits.kernel.* which sets the kernel limits.

1 Like

@tomp can you add something to the ideas board for it?

Yep will do :slight_smile: