We have built a working overlay L2 network using PeerVPN (https://peervpn.net/). This mesh VPN is deployed over a few bare metal servers running LXD.
We would like containers to see 2 NICs upon creation, the goal being for the container to be able to access both
- the internet, to expose services to the outside world
- the VPN, to securely communicate with containers hosted on other LXD servers via PeerVPN.
On each bare metal server, LXD is running the default lxdbr0 bridge in NAT mode, in addition to the tap0 virtual interface managed by PeerVPN. The tap0 interface is allocated a private IP address (192.168.X.X/32) via PeerVPN.
Each host is able to ping their peers the public IP or via the private VPN IP. Routes seems normal: default to eth0 and tap0 to access 192.168 networks.
Each container is able to access the internet and publish information to the outside world thanks to lxdbr0 and iptables filtering.
What won't work despite various tweaks ("brctl addif lxdbr0 tap0", iptables mangling and route tweaking) is the following:
- no container can ping the IP address of the local tap0 IP address
- no container can ping the IP address of another baremetal server.
We could not go any further to try and deploy a DHCP container on the VPN as they won't be able to request or allocate IP addresses. Allocating fixed 192.168.X.X/32 IP address to eth1 from within the container and adjusting routes won't solve the problem.
Ideally we would need each container to have 2 NICs eth0 and eth1:
- eth0 bridged onto "bare metal eth0", which seems to be lxdbr0 default behaviour
- eth1 bridged onto tap0, to access the VPN over which a DHCP server would be activated.
A "fallback" would be to have only 1 NIC within each container, with the routing between VPN and internet happening on the host, but that would mean extra NATing for internal VPN communications which seem pointless.
I understand this is bordeline related to LXD. Previous discussions on similar topic such as https://discuss.linuxcontainers.org/t/how-to-add-a-network-interface-in-lxc/437 won't provide a working answer so any help is appreciated.