Docker Swarm is running fine in LXD (v.3.0.1). This might sound a lot like it should be posted in a docker forum but be patient and read to the end.
We have a special case where in a docker swarm a container (docker1) has to communicate with another container (docker2) via the host (lxd) which is a LXD container.
Usually the communication within the docker swarm happens internally and directly in the docker network (docker1 talks directly with docker2).
What we need is that docker1 talks with docker2 via lxd which can be done via the exposed ports of a docker container (let me know if I should explain the exposed docker port).
So on docker1 I run
nc -vz lxd <exposed port of docker2>
On a physical machine (Ubuntu server) this works fine:
the package starts from docker1 to lxd which sends it to docker2. The answer goes from docker2 to lxd and back to docker1.
However in an LXD container (also Ubuntu) the behaviour is different:
The package starts from docker1 to lxd which sends it to docker2. So far so good. But the return path happens within the docker network from docker2 to docker1. This is a problem as there is no established connection and the package gets dropped by docker1.
We were able to work around this issue by enabling masquerading on the lxd container. But we still try to figure out why this is necessary on the LXD container but not on a physical host.
This is a tiny difference between a physical host and a LXD host regarding the network communication and docker. It is the second tiny differency that we are investigating and trying to find the root cause for. Any help is greatly appreciated!