Network ACLs with bridge networks don't work as expected

SignificantOtter · August 5, 2026, 8:17am

I have just setup an IncusOS server (version: 202608021451), which, to my understanding, includes nftables for firewall rules.

Now, according to Stéphane it should be possible by now to create ACLs within a bridge network that isolate instances against each other (since the referenced PR has been merged by now), but my attempts to do so fail in weird ways.

Here’s what I did:

Preparations

# I created a public-only ACL
$ incus network acl show remote:public-only                                                                                                     
name: public-only
description: ""
egress:
- action: allow
  state: enabled
- action: allow
  destination: 172.17.1.0/24
  state: enabled
- action: reject
  destination: 10.0.0.0/8
  state: enabled
- action: reject
  destination: 192.168.0.0/16
  state: enabled
- action: reject
  destination: 172.16.0.0/12
  state: enabled
ingress:
- action: allow
  state: enabled
config: {}
used_by: []
project: default

# I have a bridge network called incusbr0, that is attached to the default profile
$ incus network show remote:incusbr0                                           
config:
  ipv4.address: 10.90.234.1/24
  ipv4.dhcp: "true"
  ipv4.nat: "true"
  ipv6.address: fd42:1ad5:80a0:c467::1/64
  ipv6.nat: "true"
  security.acls.default.egress.action: allow
  security.acls.default.ingress.action: reject
description: Local network bridge (NAT)
name: incusbr0
type: bridge
used_by:
- /1.0/profiles/default
managed: true
status: Created
locations:
- none
project: default

# And here's the default profile
$ incus profile show remote:default                                                                                                               
config: {}
description: Default Incus profile
devices:
  eth0:
    network: incusbr0
    security.acls.default.egress.action: allow
    security.acls.default.ingress.action: allow
    type: nic
  root:
    path: /
    pool: local
    type: disk
name: default
used_by: []
project: default

Attempt 1 - network level

First, I tried to attach the ACL to the network itself:

# Attach ACL
$ incus network set remote:incusbr0 security.acls=public-only
# Create test instances
incus launch images:almalinux/10/default remote:alma-test1 --profile default
incus launch images:almalinux/10/default remote:alma-test2 --profile default
# Verify the network is used by those instances
$ incus network show remote:incusbr0
config:
  ipv4.address: 10.90.234.1/24
  ipv4.dhcp: "true"
  ipv4.nat: "true"
  ipv6.address: fd42:1ad5:80a0:c467::1/64
  ipv6.nat: "true"
  security.acls: public-only
  security.acls.default.egress.action: allow
  security.acls.default.ingress.action: reject
description: Local network bridge (NAT)
name: incusbr0
type: bridge
used_by:
- /1.0/instances/alma-test1
- /1.0/instances/alma-test2
managed: true
status: Created
locations:
- none
project: default
# Verify the ACL is used by the network
$ incus network acl show remote:public-only
name: public-only
description: ""
egress:
- action: allow
  state: enabled
- action: allow
  destination: 172.17.1.0/24
  state: enabled
- action: reject
  destination: 10.0.0.0/8
  state: enabled
- action: reject
  destination: 192.168.0.0/16
  state: enabled
- action: reject
  destination: 172.16.0.0/12
  state: enabled
ingress:
- action: allow
  state: enabled
config: {}
used_by:
- /1.0/networks/incusbr0
project: default

However, in this case, the ACL does not seem to have an effect whatsoever:

# Test if alma-test2 can be reached
$ incus exec remote:alma-test1 -- ping -c 3 10.90.234.92
PING 10.90.234.92 (10.90.234.92) 56(84) bytes of data.
64 bytes from 10.90.234.92: icmp_seq=1 ttl=64 time=0.071 ms
64 bytes from 10.90.234.92: icmp_seq=2 ttl=64 time=0.081 ms
64 bytes from 10.90.234.92: icmp_seq=3 ttl=64 time=0.083 ms

--- 10.90.234.92 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2088ms
rtt min/avg/max/mdev = 0.071/0.078/0.083/0.005 ms

# Test if outgoing traffic is possible (cloudflare IP)
$ incus exec remote:alma-test1 -- ping -c 3 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=58 time=1.43 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=58 time=1.65 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=58 time=1.53 ms

--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.426/1.536/1.652/0.092 ms

So this doesn’t work. Next on to instance/profile NIC level ACLs

Attempt 2 - ACL on instance NIC level

# Remove the network level ACL
$ incus network set remote:incusbr0 security.acls=
# Overwrite the device configuration for the test instance (the rules are egress rules, so I'm only overwriting the one where I'm running the commands)
# Note: I changed this via IncusUI, because I wasn't sure what the command would be for this
$ incus config show remote:alma-test1
architecture: x86_64
config:
# [...]
devices:
  eth0:
    network: incusbr0
    security.acls: public-only
    type: nic
ephemeral: false
profiles:
- default
stateful: false
description: ""

Now, all egress traffic seems to be blocked (not just internal):

# Test if alma-test2 can be reached
$ incus exec remote:alma-test1 -- ping -c 3 10.90.234.92
PING 10.90.234.92 (10.90.234.92) 56(84) bytes of data.

--- 10.90.234.92 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2035ms
# Test if outgoing traffic is possible (cloudflare IP)
$ incus exec remote:alma-test1 -- ping -c 3 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.

--- 1.1.1.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2076ms

What’s weirder, outgoing traffic seems to be blocked for all instances (within the bridge network?), not just the one that has the ACL attached:

# Test if alma-test1 can be reached from alma-test2
$ incus exec remote:alma-test2 -- ping -c 3 10.90.234.6
PING 10.90.234.6 (10.90.234.6) 56(84) bytes of data.
64 bytes from 10.90.234.6: icmp_seq=1 ttl=64 time=0.023 ms
64 bytes from 10.90.234.6: icmp_seq=2 ttl=64 time=0.093 ms
64 bytes from 10.90.234.6: icmp_seq=3 ttl=64 time=0.097 ms

--- 10.90.234.6 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2045ms
rtt min/avg/max/mdev = 0.023/0.071/0.097/0.033 ms

# Test if outgoing traffic is possible (cloudflare IP)
$ incus exec remote:alma-test2 -- ping -c 3 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.

--- 1.1.1.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2051ms

Just to verify, I removed the ACL and tested connectivity again:

# alma-test1 after removing the ACL:
architecture: x86_64
config:
# [...]
devices:
  eth0:
    network: incusbr0
    type: nic
ephemeral: false
profiles:
- default
stateful: false
description: ""

# Test if alma-test1 can be reached from alma-test1
$ incus exec remote:alma-test1 -- ping -c 3 10.90.234.92
PING 10.90.234.92 (10.90.234.92) 56(84) bytes of data.
64 bytes from 10.90.234.92: icmp_seq=1 ttl=64 time=0.019 ms
64 bytes from 10.90.234.92: icmp_seq=2 ttl=64 time=0.084 ms
64 bytes from 10.90.234.92: icmp_seq=3 ttl=64 time=0.082 ms

--- 10.90.234.92 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2067ms
rtt min/avg/max/mdev = 0.019/0.061/0.084/0.030 ms

# Test if outgoing traffic is possible (cloudflare IP) from alma-test1
$ incus exec remote:alma-test1 -- ping -c 3 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=58 time=1.36 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=58 time=1.56 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=58 time=1.51 ms

--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.360/1.478/1.562/0.086 ms

# Test if alma-test1 can be reached from alma-test2
$ incus exec remote:alma-test2 -- ping -c 3 10.90.234.6
PING 10.90.234.6 (10.90.234.6) 56(84) bytes of data.
64 bytes from 10.90.234.6: icmp_seq=1 ttl=64 time=0.023 ms
64 bytes from 10.90.234.6: icmp_seq=2 ttl=64 time=0.083 ms
64 bytes from 10.90.234.6: icmp_seq=3 ttl=64 time=0.083 ms

--- 10.90.234.6 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2065ms
rtt min/avg/max/mdev = 0.023/0.063/0.083/0.028 ms

# Test if outgoing traffic is possible (cloudflare IP) from alma-test2
$ incus exec remote:alma-test2 -- ping -c 3 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=58 time=1.45 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=58 time=1.50 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=58 time=1.51 ms

--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.449/1.486/1.511/0.026 ms

I must be missing something essential here, because this doesn’t make any sense to me.

stgraber · August 5, 2026, 1:07pm

The ACL not having any effect for you when applied to the bridge is normal.
When an ACL is applied to a bridge, it only acts as ingress/egress for that network’s router, it doesn’t apply to traffic that remains within the bridge (no micro-segmentation).

I’m not spotting an obvious configuration issue with the rest of the setup, so you may well be hitting a bug here. Is there anything useful in the log (incus admin os debug log -b0)?

stgraber · August 5, 2026, 8:18pm

Tracked it down to our favorite br_netfilter kernel bug for IncusOS… @gibmat

Basically the kernel gets very confused about the full traffic chain on IncusOS and decides to trigger NAT when it shouldn’t and then drop packets entirely…

Anyway, after some head banging, I’ve come up with what MAY be a working fix, doing some more testing now before I send a PR.

Basically, br_netfilter combined with our specific network interface layout in IncusOS causes traffic to get tracked multiple times, causing things to be dropped. The “fix” is to inject a “notrack” for the relevant traffic at a higher priority than the nftables nat hook.

It’s not something we can really do in Incus but it’s something we can do in IncusOS itself.

stgraber · August 5, 2026, 8:25pm

github.com/lxc/incus-os

Avoid double tracking of traffic when br_netfilter is in use (#1261)

main ← stgraber:main

opened 08:25PM - 05 Aug 26 UTC

stgraber

+68 -0

Similar to a previous issue (which we worked around in Incus instead), there are… a bunch of weird issues caused by NAT+br_netfilter effectively causing traffic to be double tracked and dropped. This workaround injects a higher priority chain into the set, then that chain gets a rule for each IncusOS bridge matching its _i* interfaces (bridge side of host veth) and marks the traffic as notrack. This then leads to the traffic being tracked exactly once and things to work correctly.

SignificantOtter · August 5, 2026, 9:02pm

@stgraber Thanks you for the replies. By now, I have found a solution for my immediate problem, using port isolation on all instances except my netbird container, which works fine for the moment.
For the future, I’m looking forward to that fix, so I can have more granular rules (e.g. selectively let some containers access a database container).

Thanks again, I love working with IncusOS so far. Everything feels clean and well integrated, without unnecessary complexity but more complex tools are available when needed (e.g. OVN, netbird, …).
Amazing work!