Prevent cross-talk

LXD
1

Thanks to: Lxd bridge doesn't work with IPv4 and UFW with nftables - #17 by tomp

I’ve got halfway where I need to be, however;

  • if I create multiple bridges, then they can talk to each other even with security.port_isolation set on both nics of the containers

how do I prevent cross-bridge cross-talk?

essentially I would want to:

  • eliminate cross-bridge communication
  • eliminate in-bridge communication
  • allow internet => container (ufw allow in on lxdbr0)
  • allow certain containers to communicate in-bridge, but those are the exception

thanks!

2

This only prevents instances on the same bridge from communicating with each other, but still allows the instances to communicate with the host, in order for traffic to be routed out onto external networks (and the internet).

In this case, “external” networks also includes any other bridges on the host system, which is why they can still communicate across bridges.

For restricting routed traffic between bridges you will need to use custom firewall rules on your host system.

We have a page on adding firewall rules to allow traffic from LXD instances, but you may find seeing the syntax for ufw useful for creating rules to block traffic between bridges.

4

Thanks for the fast response! I’ve been to that wiki article actually already, and it sadly only mentions zones for firewall-cmd, I couldn’t find a way on ufw to handle this in addition to what it mentions of letting internet => container in…

ufw doesn’t allow e.g. “ufw deny on lxdbr0 to lxdbr0” or “ufw deny on lxdbr0 to lxdbr1” so I was trying to deny based on subnet range, but that was ignored no matter what I did (incl. sorting them properly)

it would also get unwieldy if I was forced to n! for all networks I have anyway (a suggestion by stephane on the forums I’ve seen), e.g. 4 networks would already cause 24 iptable rules to deny cross-talk to each other…

would you know some solution in lxd to somehow make a default that denies them from cross-talking inside-bridges and cross-bridges and only allowing a few if I wanted to?

5

I don’t think you read until the ufw section:

6

You can probably use LXD’s own network ACL feature:

For bridge networks this cannot prevent intra-bridge communication (i.e communication between instance NICs connected to the same bridge), but you already have security.port_isolation for that. However it can prevent inter-bridge routed communication.

This sounds similar to How to stop traffic from being forwarded between lxd managed interfaces - #6 by tomp

7

I’m sorry, I should’ve mentioned ufw first to not confuse what I mean, quote with highlight:

and it sadly only mentions zones for firewall-cmd, I couldn’t find a way on ufw to handle this in addition to what it mentions of letting internet => container in…

I appreciate the ACL and forum link, however that again creates the n! complexity - the more networks I have, the more ACLs I need going both ways, correct?

Is there no LXD option to by default deny in-bridge and cross-bridge? or some way in UFW, without interrupting WAN => container?

Thank you!

8

So I’ve come up with a way to do this using a common ACL applied to each managed bridge network:

First create the managed bridge networks and connect some instances to them:

lxc network create lxdbr1
lxc network create lxdbr2
lxc launch images:ubuntu/jammy c1 -n lxdbr1
lxc launch images:ubuntu/jammy c2 -n lxdbr1
lxc launch images:ubuntu/jammy c3 -n lxdbr2
lxc launch images:ubuntu/jammy c4 -n lxdbr2

lxc network ls
+-----------------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
|      NAME       |   TYPE   | MANAGED |      IPV4       |           IPV6            | DESCRIPTION | USED BY |  STATE  |
+-----------------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
| lxdbr1          | bridge   | YES     | 10.6.53.1/24    | fd42:6775:8e08:4b29::1/64 |             | 2       | CREATED |
+-----------------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
| lxdbr2          | bridge   | YES     | 10.1.70.1/24    | fd42:59cb:e895:e486::1/64 |             | 2       | CREATED |
+-----------------+----------+---------+-----------------+---------------------------+-------------+---------+---------+

lxc ls
+--------+---------+--------------------+-----------------------------------------------+-----------+-----------+
|  NAME  |  STATE  |        IPV4        |                     IPV6                      |   TYPE    | SNAPSHOTS |
+--------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| c1     | RUNNING | 10.6.53.254 (eth0) | fd42:6775:8e08:4b29:216:3eff:fefa:9454 (eth0) | CONTAINER | 0         |
+--------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| c2     | RUNNING | 10.6.53.178 (eth0) | fd42:6775:8e08:4b29:216:3eff:feae:2263 (eth0) | CONTAINER | 0         |
+--------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| c3     | RUNNING | 10.1.70.10 (eth0)  | fd42:59cb:e895:e486:216:3eff:fecb:b062 (eth0) | CONTAINER | 0         |
+--------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| c4     | RUNNING | 10.1.70.110 (eth0) | fd42:59cb:e895:e486:216:3eff:feed:891a (eth0) | CONTAINER | 0         |
+--------+---------+--------------------+-----------------------------------------------+-----------+-----------+

Now prevent intra-bridge cross-talk using port-isolation:

lxc config device set c1 eth0 security.port_isolation=true
lxc config device set c2 eth0 security.port_isolation=true
lxc config device set c3 eth0 security.port_isolation=true
lxc config device set c4 eth0 security.port_isolation=true

Test intra-bridge cross talk blocked:

lxc exec c1 -- ping c2 -4 -c1
PING  (10.6.53.178) 56(84) bytes of data.
From c1.lxd (10.6.53.254) icmp_seq=1 Destination Host Unreachable

---  ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

Test inter-bridge cross talk still allowed:

lxc exec c1 -- ping 10.1.70.10 -c1
PING 10.1.70.10 (10.1.70.10) 56(84) bytes of data.
64 bytes from 10.1.70.10: icmp_seq=1 ttl=63 time=0.072 ms

--- 10.1.70.10 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.072/0.072/0.072/0.000 ms

Now setup ACL to prevent inter-bridge routed cross-talk:

lxc network set lxdbr1 \
    security.acls.default.egress.action=allow \
    security.acls.default.ingress.action=allow
lxc network set lxdbr2 \
    security.acls.default.egress.action=allow \
    security.acls.default.ingress.action=allow

lxc network acl create external-only
Network ACL external-only created

lxc network acl rule add external-only egress \
    destination=10.6.53.0/24 \
    action=reject
lxc network acl rule add external-only egress \
    destination=10.1.70.0/24 \
    action=reject

lxc network acl show external-only
name: external-only
description: ""
egress:
- action: reject
  destination: 10.6.53.0/24
  state: enabled
- action: reject
  destination: 10.1.70.0/24
  state: enabled
ingress: []
config: {}
used_by: []

Now apply the ACL to each network:

lxc network set lxdbr1 security.acls=external-only
lxc network set lxdbr2 security.acls=external-only

Test inter-bridge routed traffic blocked:

lxc exec c1 -- ping 10.1.70.10 -c1
PING 10.1.70.10 (10.1.70.10) 56(84) bytes of data.
From 10.6.53.1 icmp_seq=1 Destination Port Unreachable

--- 10.1.70.10 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

Test external traffic still allowed (and that DNS is working):

lxc exec c1 -- ping www.linuxcontainers.org -4 -c1
PING rproxy.dcmtl.stgraber.org (45.45.148.7) 56(84) bytes of data.
64 bytes from rproxy.dcmtl.stgraber.org (45.45.148.7): icmp_seq=1 ttl=51 time=91.4 ms

--- rproxy.dcmtl.stgraber.org ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 91.423/91.423/91.423/0.000 ms
5 Likes
9

You’re truly incredible, I tried to play around with ACL myself, but never thought of doing it this simple to achieve the goal, thank you so much!

Just wondering, is there a way by default to assign this ACL? or would I have to do this for each new bridge:

lxc network set xxx \
    security.acls.default.egress.action=allow \
    security.acls.default.ingress.action=allow

and

lxc network set xxx security.acls=external-only

Thanks again! :smile:

1 Like
10

Thanks!

Yes you’d need to do those steps for each new network as there isn’t the concept of “profiles” for networks currently.

You should be able to do it in one step though, e.g.

lxc network set xxx \
    security.acls.default.egress.action=allow \
    security.acls.default.ingress.action=allow \
    security.acls=external-only

And then add the new network’s subnet to the external-only ACL.

1 Like
11

oh right, I can just chain them, that’s fine; thank you again for it all!! :smile: