I am back to this. I thought that I had developed a solution to this but it seems to not be as flexible as I had hoped.
To recap, our nodes have two LANs, say LAN-1 and LAN-2. The containers are set up with their own IP addresses that we can say are in LAN-3.
The needs are:
- We want to be able to assign a static IP in LAN-3 to the containers, and be able to reach them from either LAN-1 or LAN-2.
- We want to be able to easily migrate a container from HOST A to HOST B and preserve routability.
Based on the thread above, we’ve set up every machine with a bridge that is not connected to either network called lxdbr0. It has a static IP address of 192.168.122.1 on all nodes. Inside the container, each container has its own network set up, own IP address, and it has a default route to 192.168.122.1.
The challenge is building a route on the host node to the container.
We’ve tried two techniques:
- lxc network set lxdbr0 ipv4.routes CONTAINER-IP
– This sets up the route properly, but as new containers are added and others moved or deleted, we need to rebuild the entire route list. Ie., if I have container 1,2,3 on the host, and add container4, I need to run:
lxc network set lxdbr0 ivp4.routes “1,2,3,4”
If I move 3 to another host, I need to then run:
lxc network set lxdbr0 ipv4.routes “1,2,4”
Then I need to flush the routing table to get rid of the route to 3.
- a simple host route to the bridge:
ip route add 3 dev lxdbr0
The drawback to this is that the route needs to be added and removed from both ends each time we move a container.
We’re looking for the ideal–or recommended–way to be able to build and tear-down these container-specific routes when the containers are started, stopped or moved.