Isolation of LXD containers?

LXD
1

I’m a relatively recent user of LXD, and am still exploring. I’m liking its power and flexibility so far, so thanks to the developers!

I’ve been looking at using LXD to create isolated containers (a DMZ of sorts) for services that I need to give external network access. I’d like to stress-test whether my approach makes sense.

On my router I have various VLANs. My main internal network is VLAN 10, IPv4 subnet 10.10.10.0/24. I have also created a VLAN 66, IPv4 subnet 172.16.66.0/24, for the proposed isolated containers.

On the router, I have set up strict firewall rules (essentially “guest” network rules) so that any host on VLAN 66 cannot reach any other host in the network, except for limited exceptions (eg DHCP, DNS, ICMPv6, and certain established/related connections). However, hosts on VLAN 10 can reach VLAN 66.

My LXD host (Ubuntu 18.04.3) is on subnet 10.10.10.0/24. Its NIC has both VLAN 10 (native) and VLAN 66 (tagged) switched to it.

I created an Ubuntu 20.04 container and then followed Simos’ very helpful guide to set up a macvlan profile. I added VLAN 66 to that profile by following this post as well.

The result is that the container gets IPv4 and IPv6 addresses in VLAN 66 (only), and is accessible from other hosts in my network (including the LXD host) in accordance with my router’s firewall rules. (I also mounted a storage directory on my RAID6 array into the container using the LXD device option.)

This seems to achieve what I wanted - I have a container that has addresses in VLAN 66 (only) and is isolated by my router from other hosts in the network. I can forward ports on my router directly to the container.

I wanted to check though that I wasn’t deceiving myself, in particular whether LXD’s networking in some way opens up the rest of the network to the container in a way that I thought I had shut off through the router’s firewall rules.

Any insights appreciated!

2

Hi, and welcome.

You’re thorough description of your setup leads me to think that you’ve set it up exactly right for your needs.

Using the macvlan NIC type (options documented here https://linuxcontainers.org/lxd/docs/master/instances#nictype-macvlan) combined with the vlan option will create a VLAN alias interface for your physical interface on your host, and then a macvlan interface inside your container on top of it.

The fact that you are only getting IPs from the desired VLAN suggests its working well.

From a security perspective, macvlan NIC types by design prevent the container communicating with the host as well.

One thing to be aware of though is that macvlans effectively join the container onto the parent network (in this case your VLAN interface) at layer 2. This means that your containers will be able to directly communicate with each other and any other host on the same VLAN (without going through your router’s firewall). They can also send broadcast packets to that VLAN and so can ‘take’ any IP address on that VLAN (via ARP), potentially allowing them to interfere with or pretend to be other IPs on the network.

But because traffic to reach the other VLANs has to go through your router’s firewall, then those rules take effect at that point.

3

Thanks for the welcome and the very quick reply!

Your info is exactly what i was after - especially the clarification in the second last para.

So I don’t quite have it how I want. I want each “VLAN 66 container” isolated from each other. That suggests I need a separate VLAN for each, am i right? Or is there another, better way?

Thanks again.

4

I should add that I need the LXD host to be able to access the containers. Thanks

5

Sorry for the stream of consciousness here, but I am researching and learning as I go.

Am I right in thinking that I could achieve what I want (each VLAN 66 container isolated from each other) simply by relevant firewall rules on the LXD host, since the LXD host is (as I understand it) acting as the L2 switch? (Although that wouldn’t solve the IP spoofing issue I suppose.)

If firewall rules are what I need, I have a couple of follow-on questions:

  • Am I right in setting up the macvlan profile in LXD on my physical interface (enp0s31f6) and not on the bridge (br0) that the physical interface is enslaved to? I have a legacy bridge from when I ran KVM VMs, and am actually currently using that bridge for another LXD container that I want as a fully fledged member of VLAN 10, subnet 10.10.10.0/24.

  • Instead of using macvlan, could I achieve what I want with another bridge that is set up in netplan to use VLAN 66 (plus with the firewall rules on the LXD host, including to isolate the host)? If so, is there any advantage to using one approach (bridge vs macvlan) over the other?

Thanks again for the help.

Edit: Maybe something along these lines is what I need?

6

I’ve managed to achieve a solution that I think is right (it so far seems to work OK…). There are probably several ways of achieving a similar outcome but this is what I settled on (outlining it here in case someone else finds it useful).

First, I created a new Linux bridge in netplan for VLAN 66:

network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s31f6:
      dhcp4: false
      dhcp6: false
  bridges:
    br0:
      interfaces: [ enp0s31f6 ]
      dhcp4: true
      dhcp6: true   
      ipv6-privacy: true
      parameters:
        forward-delay: 0
        stp: false
    br66:
      interfaces: [ vlan66 ]
      dhcp4: false
      dhcp6: false
      link-local: [ ]
      accept-ra: false
      parameters:
        forward-delay: 0
        stp: false
  vlans:
    vlan66:
      id: 66
      link: enp0s31f6
      dhcp4: false
      dhcp6: false

I could have simply created the VLAN 66 interface in netplan and used LXD to create the bridge using that interface - with the bridge being “public” rather than “private”, by ensuring that LXD did not create any addresses or set up NAT or DHCP - a bit like described here. I actually tested that configuration out and it did work equivalently. However, I decided for flexibility to create the bridge outside LXD.

Note that I deliberately set up the bridge (br66) to not receive any IP address - not IPv4, not IPv6, not even link-local. It’s purely intended as the interface into the VLAN 66 containers.

Next I created a device profile in LXD using that bridge:

devices:
  eth0:
  nictype: bridged
  parent: br66
  security.mac_filtering: "true"
  type: nic

Then I enabled the br_netfilter kernel module and relevant kernel parameters (as described here) and finally set up a bunch of iptables rules (mainly in the FORWARD chain) to regulate traffic flow - principally to ensure that VLAN 66 containers couldn’t generally reach other internal hosts. Given the various interfaces and networks in my setup, this was probably the hardest bit!

Of course by not having LXD manage the bridge, I can’t currently secure it with security.ipv4_filtering or security.ipv6_filtering, but hopefully that will be changed soon, as indicated here.

7

Hi. Yes that approach is probably a lot simpler than having a vlan per container.

And as you say, by using firewall on the LXD host to enforce your network policy with regard to source address that works too.

I made some changes to the bridge driver recently that should make allowing IP filtering on unmanaged bridges easier to implement, so its still something that is on my radar.