Solution
See Nictype: bridged and network isolation via pfSense - #2 by thetredev
The plan
I would like to block any traffic (at first) between Incus containers which have bridged NICs attached to them, going through a pfSense instance which acts as a gateway and firewall for those containers.
Currently, I can block everything using pfSense except direct communication between containers, like ping 10.250.0.2 from 10.250.0.3. My setup and further details are explained below.
The setup
I’m running an IPv4 only setup on a single Incus host:
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
$ uname -a
Linux incus 6.13.11-zabbly+ #debian12 SMP PREEMPT_DYNAMIC Thu Apr 10 21:24:02 UTC 2025 x86_64 GNU/Linux
$ incus version
Client version: 6.11
Server version: 6.11
Both Incus and the kernel are installed from Zabbly repos, with ZFS as Incus storage also provided by Zabbly repos.
Incus
Networking
$ incus network list
+------------+----------+---------+-------------+------+-------------+---------+---------+
| NAME | TYPE | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY | STATE |
+------------+----------+---------+-------------+------+-------------+---------+---------+
| enp7s0 | physical | NO | | | | 1 | |
+------------+----------+---------+-------------+------+-------------+---------+---------+
| lan0 | bridge | YES | 10.0.0.1/16 | none | | 10 | CREATED |
+------------+----------+---------+-------------+------+-------------+---------+---------+
| lan0-proxy | bridge | YES | none | none | | 6 | CREATED |
+------------+----------+---------+-------------+------+-------------+---------+---------+
| lo | loopback | NO | | | | 0 | |
+------------+----------+---------+-------------+------+-------------+---------+---------+
| wan0 | bridge | NO | | | | 3 | |
+------------+----------+---------+-------------+------+-------------+---------+---------+
The relevant interfaces are listed in the following table:
| Name | IPv4 CIDR | Managed by | Usage | Instance communication allowed? |
|---|---|---|---|---|
| lan0 | 10.0.0.1/16 | Incus | Internal network for internal instances. | YES |
| lan0-proxy | 10.127.0.0/16 | Incus: no gateway address needed. | Internal network for proxy communication (HTPS/ACME). | YES |
| wan0 | 10.250.0.1/16 | pfSense instance: IP address is set within pfSense. | Internal network for internal instances, pfSense acts as default gateway for those | NO |
Instances
$ incus list -c n4td
+------------------------------+------------------------------+-----------------+----------------------------------------------+
| NAME | IPV4 | TYPE | DESCRIPTION |
+------------------------------+------------------------------+-----------------+----------------------------------------------+
| 000-base | | CONTAINER | Template container |
| | | | |
| | | | realm: internal, proxy |
| | | | runtime: LXC |
| | | | |
+------------------------------+------------------------------+-----------------+----------------------------------------------+
| 000-lan-proxy | 10.127.0.1 (eth1) | CONTAINER | Caddy ACME proxy |
| | 10.0.1.0 (eth0) | | |
| | | | realm: internal, proxy |
| | | | runtime: LXC |
| | | | |
+------------------------------+------------------------------+-----------------+----------------------------------------------+
| 001-ns | 10.127.0.4 (eth1) | CONTAINER | Technitium DNS: DNS/DHCP appliance |
| | 10.0.0.2 (eth0) | | |
| | | | realm: internal, proxy |
| | | | runtime: LXC |
| | | | |
+------------------------------+------------------------------+-----------------+----------------------------------------------+
| 002-pf | REDACTED | VIRTUAL-MACHINE | pfSense: manage external services |
| | | | |
| | | | realm: external |
| | | | |
+------------------------------+------------------------------+-----------------+----------------------------------------------+
| 004-gitlab-infra | 172.18.0.1 (br-a9d7ca15e238) | CONTAINER | GitLab: infra deployments |
| | 172.17.0.1 (docker0) | | |
| | 10.127.0.8 (eth1) | | realm: internal, proxy |
| | 10.0.1.1 (eth0) | | runtime: LXC/Docker |
| | | | |
+------------------------------+------------------------------+-----------------+----------------------------------------------+
| 005-registry-infra | 172.19.0.1 (br-cf0fdaca15e0) | CONTAINER | Sonatype Nexus3: infra registry/proxy/mirror |
| | 172.18.0.1 (br-3ae86106b6b1) | | |
| | 172.17.0.1 (docker0) | | realm: internal, proxy |
| | 10.127.0.2 (eth1) | | runtime: LXC/Docker |
| | 10.0.1.4 (eth0) | | |
+------------------------------+------------------------------+-----------------+----------------------------------------------+
| 021-external-ts3-test | 172.18.0.1 (br-f505729aba7b) | CONTAINER | TeamSpeak 3: public test |
| | 172.17.0.1 (docker0) | | |
| | 10.250.0.2 (eth0) | | realm: external |
| | | | runtime: LXC/Docker |
| | | | |
+------------------------------+------------------------------+-----------------+----------------------------------------------+
| 061-external-css-pub-01-test | 10.250.0.3 (eth0) | CONTAINER | Counter-Strike: Source [pub01] test |
| | | | |
| | | | realm: external |
| | | | runtime: LXC |
| | | | |
+------------------------------+------------------------------+-----------------+----------------------------------------------+
pfSense / Managing external services
I want to manage the networking for the services/applications provided by the two instances 061-external-css-pub-01-test and 021-external-ts3-test via pfSense only. The whole setup surrounding wan0 interface is my first attempt at this.
I have tried to do the following:
$ cat /etc/network/interfaces
# REDACTED
auto wan0
iface wan0 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
#SVC_PUB
$ incus config show 002-pf
REDACTED
devices:
REDACTED
eth2:
nictype: bridged
parent: wan0
type: nic
# note: IP address 10.250.0.1/16 is set for eth2 within pfSense
$ incus config show 021-external-ts3-test
REDACTED
devices:
eth0:
nictype: bridged
parent: wan0
type: nic
$ incus exec 021-external-ts3-test cat -- /etc/systemd/network/eth0.network
[Match]
Name=eth0
[Network]
Address=10.250.0.2/16
Gateway=10.250.0.1
DNS=8.8.8.8
$ incus exec 021-external-ts3-test -- ip a show dev eth0
82: eth0@if83: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 10:66:6a:14:e8:5b brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.250.0.2/16 brd 10.250.255.255 scope global eth0
valid_lft forever preferred_lft forever
$ incus config show 061-external-css-pub-01-test
REDACTED
devices:
eth0:
nictype: bridged
parent: wan0
type: nic
$ incus exec 061-external-css-pub-01-test -- cat /etc/systemd/network/eth0.network
[Match]
Name=eth0
[Network]
Address=10.250.0.3/16
Gateway=10.250.0.1
DNS=8.8.8.8
$ incus exec 061-external-css-pub-01-test -- ip a show dev eth0
84: eth0@if85: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 10:66:6a:47:ff:bb brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.250.0.3/16 brd 10.250.255.255 scope global eth0
valid_lft forever preferred_lft forever
The problem
Within pfSense, eth2’s firewall rules are currently NOT set, meaning no traffic is allowed at all, as DROP (or REJECT?) is the default rule. This almost works. From within any of the 2 test containers, I cannot ping anything internal or external except other instances besides pfSense. I don’t understand why.
As soon as I put a firewall rule in place to allow ICMP to 10.250.0.1 (pfSense itself), pinging that address works for both contains. So the packets are indeed going through pfSense. But that rule only additionally allows pinging to wan0, pinging the containers works regardless, even if I explicitly put a block ALL rule as the sole rule for eth2.
How can I truly isolate container communications using this kind of setup? I want to manage firewall rules for external services via pfSense, and anything internal via Incus (and ACLs). Incus should not mess with anything going through wan0, and pfSense should not mess with anything that Incus manages either. How to achieve this?
Thanks!