Docker Swarm is running fine in LXD (v.3.0.1). This might sound a lot like it should be posted in a docker forum but be patient and read to the end.
We have a special case where in a docker swarm a container (docker1) has to communicate with another container (docker2) via the host (lxd) which is a LXD container.
Usually the communication within the docker swarm happens internally and directly in the docker network (docker1 talks directly with docker2).
What we need is that docker1 talks with docker2 via lxd which can be done via the exposed ports of a docker container (let me know if I should explain the exposed docker port).
So on docker1 I run nc -vz lxd <exposed port of docker2>
On a physical machine (Ubuntu server) this works fine:
the package starts from docker1 to lxd which sends it to docker2. The answer goes from docker2 to lxd and back to docker1.
However in an LXD container (also Ubuntu) the behaviour is different:
The package starts from docker1 to lxd which sends it to docker2. So far so good. But the return path happens within the docker network from docker2 to docker1. This is a problem as there is no established connection and the package gets dropped by docker1.
We were able to work around this issue by enabling masquerading on the lxd container. But we still try to figure out why this is necessary on the LXD container but not on a physical host.
This is a tiny difference between a physical host and a LXD host regarding the network communication and docker. It is the second tiny differency that we are investigating and trying to find the root cause for. Any help is greatly appreciated!
Finally I was able to find the cause why docker didn’t run really smooth in our LXD containers.
The host system needs libvirt-bin and qemu-kvm installed. This resolves every issue we had.
I discovered it because docker was running smoothly on a new server that was set up freshly but still behaving on our old servers. So I compared the configs of the two which should have been the same. Turned out those two packages were missing. I assume that they are an install dependency on the latest LXD version but were not on LXD 2.x? Not sure. But happy that it’s finally running!
Hi there.
I am facing similar challenges.
I’ve installed libvirt-bin and qemu-kvm, both on the host and inside the LXD containers but still …
I keep getting the below error:
error=“error creating external connectivity network: cannot restrict inter-container communication: please ensure that br_netfilter kernel module is loaded”
Can someone help me sort it out ?
I am running LXD 3.0.1
I am not familiar with the whole setup of Docker Swarm in LXD. However, the message you get is quite straightforward. The container needs to have the br_netfilter kernel module loaded on the host (or parent VM), but cannot load it itself. You should load the kernel module on the host and try again.
If this solves the issue, then you can configure either the host to always load this kernel module, or the container config to pre-load this kernel module when the container starts (see How to add kernel modules into an LXC container? - #3 by simos).
Thanks Simos,
However I had checked that before: I do have the br_netfilter loaded by default in the host. (I checked with lsmod | grep br_netfilter) and I even tried to load it explicitly with:
lxc config set my_lxd_container linux.kernel_modules br_netfilter
But the error persists …