Isolated containers and haproxy

I’m new to containers and I’m trying to set this up in my home lab as a test for future projects. I’ve been digging via google and several searches in the forums for a situation that is similar to what i’m looking to accomplish. I can’t seem to wrap my head around what I need to do in order to get this going. I have 1 Ubuntu18 Host and I want to run containers inside that host.

Here’s a very simple diagram of my setup.

WAN / Firewall
–Host (LAN) device id: [ens160]
----C1 (HAProxy & SSH)
----C2 (Apache & SSH)
----C3 (Apache & SSH)

I want all of the containers to have regular outgoing internet access via my host (NAT MASQ).
I want to forward incoming host ports 80 and 443 to C1.
I want C1 to have full connectivity to C2 & C3 for HAProxy to connect to their services.
I do not want C2 & C3 to be able talk to each other at all. No network communication at all.
I do not want any of my containers to be able to talk to anything on the host network (LAN).
I want to give SSH access to each container from C1 and possibly from the internet on different ports.

My host has 1 network card behind an existing firewall. I plan to forward whatever ports needed for this from my firewall to the host. I’ve played around with the bridging, macvlan, and several of the network types. I can’t seem to get this right. I want to isolate each container so the only thing the container can talk to is C1.

Is this out of scope for LXD? Would docker make a better fit for this? From what I’ve read if you are using ssh inside your containers docker isn’t a good fit as a solution.

Any help is appreciated.

Maybe you could wait until LXD 3.19 is out (IIRC it was supposed to be available at end of last year so it should not be a too long wait) and try the new ‘routed’ nictype ? It looks a bit like your use case, maybe @tomp could comment on this idea ?

I use shorewall to manage the network access between containers/host/internet.
I’m not sure if it can do all the things you want. I think it can.
I use it mostly to forward ports.
To configure shorewall, start with the 2-interface configuration template.

Hi,

So you should be able to achieve this, but you’ll need to add some custom firewall rules.

If you use an LXD bridge network, and connect each container to it, this will achieve the outbound NAT you’re looking for.

Then if you give each of the container’s a static IP using lxc config device set c1 eth0 ip.address=x.x.x.x then dnsmasq will allocate a static IP via DHCP on each boot.

You can then use the bridge IP filtering to stop the containers hijacking other’s IPs if you like.
See https://linuxcontainers.org/lxd/docs/master/instances#nictype-bridged

You can then use either DNAT iptables rules or the LXD proxy to forward local ports on the host to C1.
See proxy device for more details: https://linuxcontainers.org/lxd/docs/master/instances#type-proxy

Finally you would add iptables rules on the host to restrict traffic from the other 2 containers as you need.

Be sure to load br_netfilter kernel module and enable the following sysctls so that packets traversing the bridge are passed through iptables though.

  net.bridge.bridge-nf-call-arptables = 1
  net.bridge.bridge-nf-call-ip6tables = 1
  net.bridge.bridge-nf-call-iptables = 1

This should get you want you need.

As @gpatel-fr suggested you could also achieve similar using the forthcoming routed nic type. However you’d still need to use custom firewall rules, all this really does in this case is avoid the need for using the IP filtering feature on the bridged NIC (as its inherent on routed NIC types).

Duh, I tried it and it works. Good stuff.