Hello!
I am trying to isolate the traffic between two of my subnets (each created with LXD), however no matter what I try, I am still able to ping one of them from a container on the other network. I have even tried creating an unmanaged network and iptables rules which block the subnet from sending traffic to the other one, yet pings are still able to go through.
What steps should I take to completely isolate the subnets so that they are not able to reach each other, however can still have access to the internet as bridges?
Thank you in advance.
Please can you show the output of ip a and ip r on the host and inside a container connected to each subnet.
Please also show the output of lxc network show <network> for each network, so I can get a better idea of your configuration.
Host
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether [Removed] brd ff:ff:ff:ff:ff:ff
inet [Removed]/22 brd [Removed] scope global eth0
valid_lft forever preferred_lft forever
inet6 [Removed]/64 scope global
valid_lft forever preferred_lft forever
inet6 [Removed]/64 scope link
valid_lft forever preferred_lft forever
3: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:bc:2b:6f brd ff:ff:ff:ff:ff:ff
inet 10.10.0.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:febc:2b6f/64 scope link
valid_lft forever preferred_lft forever
5: vethf572811e@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether c2:22:49:16:47:88 brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: veth453beee5@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 36:f7:34:fe:0a:bd brd ff:ff:ff:ff:ff:ff link-netnsid 1
9: veth718b5b5d@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 2e:34:03:65:89:44 brd ff:ff:ff:ff:ff:ff link-netnsid 2
11: veth4de8a656@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether ba:7b:21:2e:84:2b brd ff:ff:ff:ff:ff:ff link-netnsid 3
13: veth74fe75ac@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 06:f3:e7:ed:8b:9c brd ff:ff:ff:ff:ff:ff link-netnsid 4
15: veth1b4c8e81@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 96:c2:7d:5f:30:a1 brd ff:ff:ff:ff:ff:ff link-netnsid 5
17: vethaded18eb@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 5a:96:3b:1d:3d:21 brd ff:ff:ff:ff:ff:ff link-netnsid 6
19: vethc4ceee56@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether ea:31:a5:a4:58:07 brd ff:ff:ff:ff:ff:ff link-netnsid 7
21: veth85220e80@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 72:87:b1:33:2e:6d brd ff:ff:ff:ff:ff:ff link-netnsid 8
23: veth4b5fc252@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 82:4f:d5:0a:08:ec brd ff:ff:ff:ff:ff:ff link-netnsid 9
25: vethba4896a7@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 72:40:02:08:f6:b6 brd ff:ff:ff:ff:ff:ff link-netnsid 10
27: veth377d5e63@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether a6:dc:90:95:f2:f1 brd ff:ff:ff:ff:ff:ff link-netnsid 11
29: veth87441456@if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 16:b7:f8:0c:65:f8 brd ff:ff:ff:ff:ff:ff link-netnsid 13
31: veth24404bed@if30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 52:34:00:10:c5:9c brd ff:ff:ff:ff:ff:ff link-netnsid 14
33: veth05e8c61b@if32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 06:a7:08:b8:82:2c brd ff:ff:ff:ff:ff:ff link-netnsid 15
35: veth8c1b3288@if34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 32:df:11:b7:eb:ab brd ff:ff:ff:ff:ff:ff link-netnsid 16
37: vethe9e1efcc@if36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 72:9f:52:64:bb:1b brd ff:ff:ff:ff:ff:ff link-netnsid 17
39: vethf807a990@if38: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 06:05:88:be:85:dc brd ff:ff:ff:ff:ff:ff link-netnsid 18
40: lxdbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:3f:6c:bc brd ff:ff:ff:ff:ff:ff
inet 10.10.1.1/24 scope global lxdbr1
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe3f:6cbc/64 scope link
valid_lft forever preferred_lft forever
42: veth3b18c09c@if41: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr1 state UP group default qlen 1000
link/ether a6:b4:fe:6e:63:82 brd ff:ff:ff:ff:ff:ff link-netnsid 19
Apologies for the long list above; I’m running quite a few containers on the host at the moment.
$ ip r
default via [Removed] dev eth0 proto static
10.10.0.0/24 dev lxdbr0 proto kernel scope link src 10.10.0.1
10.10.1.0/24 dev lxdbr1 proto kernel scope link src 10.10.1.1
[Removed]/22 dev eth0 proto kernel scope link src [Removed]
Container on subnet 1
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
38: eth0@if39: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:11:d8:10 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.10.0.123/24 brd 10.10.0.255 scope global dynamic eth0
valid_lft 3041sec preferred_lft 3041sec
inet6 fe80::216:3eff:fe11:d810/64 scope link
valid_lft forever preferred_lft forever
$ ip r
default via 10.10.0.1 dev eth0 proto dhcp src 10.10.0.123 metric 100
10.10.0.0/24 dev eth0 proto kernel scope link src 10.10.0.123
10.10.0.1 dev eth0 proto dhcp scope link src 10.10.0.123 metric 100
Container on subnet 2
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
41: eth0@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:01:48:86 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.10.1.2/24 brd 10.10.1.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe01:4886/64 scope link
valid_lft forever preferred_lft forever
$ ip r
default via 10.10.1.1 dev eth0 proto static
10.10.1.0/24 dev eth0 proto kernel scope link src 10.10.1.2
Network configuration for subnet 1 (lxdbr0)
$ lxc network show lxdbr0
config:
ipv4.address: 10.10.0.1/24
ipv4.firewall: “false”
ipv4.nat: “false”
ipv6.address: none
description: “”
name: lxdbr0
type: bridge
managed: true
status: Created
locations:
- none
Network configuration for subnet 2 (lxdbr1)
$ lxc network show lxdbr1
config:
ipv4.address: 10.10.1.1/24
ipv4.firewall: “false”
ipv4.nat: “false”
ipv6.address: none
description: “”
name: lxdbr1
type: bridge
managed: true
status: Created
locations:
- none
Please note that I’ve removed the used_by sections from each network configuration. I have also implemented the required iptables rules into my own script and have therefore disabled LXD’s firewall and NAT features on each network.
What iptables rules have you added?
These are the rules I have added to the host system. None of my containers have any rules on them.
-A INPUT -i lxdbr0 -p tcp -m tcp --dport 53 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 53 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 67 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A INPUT -i lxdbr1 -p tcp -m tcp --dport 53 -m comment --comment “generated for LXD network lxdbr1” -j ACCEPT
-A INPUT -i lxdbr1 -p udp -m udp --dport 53 -m comment --comment “generated for LXD network lxdbr1” -j ACCEPT
-A INPUT -i lxdbr1 -p udp -m udp --dport 67 -m comment --comment “generated for LXD network lxdbr1” -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A FORWARD -o lxdbr0 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A FORWARD -i lxdbr1 -m comment --comment “generated for LXD network lxdbr1” -j ACCEPT
-A FORWARD -o lxdbr1 -m comment --comment “generated for LXD network lxdbr1” -j ACCEPT
-A OUTPUT -o lxdbr0 -p tcp -m tcp --sport 53 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 53 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 67 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A OUTPUT -o lxdbr1 -p tcp -m tcp --sport 53 -m comment --comment “generated for LXD network lxdbr1” -j ACCEPT
-A OUTPUT -o lxdbr1 -p udp -m udp --sport 53 -m comment --comment “generated for LXD network lxdbr1” -j ACCEPT
-A OUTPUT -o lxdbr1 -p udp -m udp --sport 67 -m comment --comment “generated for LXD network lxdbr1” -j ACCEPT
-A POSTROUTING -s 10.10.0.0/24 ! -d 10.10.0.0/24 -m comment --comment “generated for LXD network lxdbr0” -j MASQUERADE
-A POSTROUTING -s 10.10.1.0/24 ! -d 10.10.1.0/24 -m comment --comment “generated for LXD network lxdbr1” -j MASQUERADE
I cannot see any DROP rules in that ruleset, have you changed the default policy to drop?
My apologies. I only sent my LXD-related rules. My default policy is drop on both the input and forward chains and accept on the output chain.
And what is the output of:
sudo sysctl -a | grep net.bridge.bridge-nf-call
On the host there’s no output.
OK so you’ve not got the bridge filtering mode enabled, which is why your iptables rules are not taking effect.
You need to load the kernel module br_netfilter and then ensure those sysctls are enabled.
More info here
Am I right in thinking that it’s as simple as running the following?
sudo modprobe br_netfilter
echo 1 | sudo tee /proc/sys/net/bridge/bridge-nf-call-iptables
Yes thats right, in fact I believe it defaults those settings to enabled when the module is loaded.
I’ve run those two commands yet pings are still traversing subnets. Attached below are my iptables rule chains.
Output of iptables -S
-P INPUT DROP
-P FORWARD DROP
-P OUTPUT ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -i lxdbr0 -p tcp -m tcp --dport 53 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 53 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 67 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A INPUT -i lxdbr1 -p tcp -m tcp --dport 53 -m comment --comment “generated for LXD network lxdbr1” -j ACCEPT
-A INPUT -i lxdbr1 -p udp -m udp --dport 53 -m comment --comment “generated for LXD network lxdbr1” -j ACCEPT
-A INPUT -i lxdbr1 -p udp -m udp --dport 67 -m comment --comment “generated for LXD network lxdbr1” -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A FORWARD -o lxdbr0 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A FORWARD -i lxdbr1 -m comment --comment “generated for LXD network lxdbr1” -j ACCEPT
-A FORWARD -o lxdbr1 -m comment --comment “generated for LXD network lxdbr1” -j ACCEPT
-A OUTPUT -o lxdbr0 -p tcp -m tcp --sport 53 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 53 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 67 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A OUTPUT -o lxdbr1 -p tcp -m tcp --sport 53 -m comment --comment “generated for LXD network lxdbr1” -j ACCEPT
-A OUTPUT -o lxdbr1 -p udp -m udp --sport 53 -m comment --comment “generated for LXD network lxdbr1” -j ACCEPT
-A OUTPUT -o lxdbr1 -p udp -m udp --sport 67 -m comment --comment “generated for LXD network lxdbr1” -j ACCEPT
Output of iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -s 10.10.0.0/24 ! -d 10.10.0.0/24 -m comment --comment “generated for LXD network lxdbr0” -j MASQUERADE
-A POSTROUTING -s 10.10.1.0/24 ! -d 10.10.1.0/24 -m comment --comment “generated for LXD network lxdbr1” -j MASQUERADE
What ping command?
This is a ping going from a container on one network to a container on another network.
$ ping 10.10.0.123
PING 10.10.0.123 (10.10.0.123) 56(84) bytes of data.
64 bytes from 10.10.0.123: icmp_seq=1 ttl=63 time=0.089 ms
64 bytes from 10.10.0.123: icmp_seq=2 ttl=63 time=0.115 ms
64 bytes from 10.10.0.123: icmp_seq=3 ttl=63 time=0.147 ms
64 bytes from 10.10.0.123: icmp_seq=4 ttl=63 time=0.159 ms
^C
— 10.10.0.123 ping statistics —
4 packets transmitted, 4 received, 0% packet loss, time 3078ms
rtt min/avg/max/mdev = 0.089/0.127/0.159/0.027 ms
These rules allow all forwarded traffic to/from your subnets:
Pings are still going through with these as my forward rules. Am I doing something wrong here?
-A FORWARD -i lxdbr0 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A FORWARD -o lxdbr0 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A FORWARD -s 10.10.1.0/24 -d 10.10.0.0/24 -j DROP
lxdbr0 is 10.10.0.0/24
lxdbr1 is 10.10.1.0/24
These rules would allow all traffic to/from lxdbr0:
-A FORWARD -i lxdbr0 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
-A FORWARD -o lxdbr0 -m comment --comment “generated for LXD network lxdbr0” -j ACCEPT
You need to remove those and add replacement rules to only allow the traffic you want.
My apologies. I was under the impression that I could simply overwrite them by appending rules as a “blacklist” of sorts.
The rules are processed in order from first to last, any rule jumping -j to the ACCEPT target is allowed.