Can someone clarify how network subject selectors like @internal and @external are intended to work in ACLs? I had assumed that @internal would match sources and destinations within the Incus network, while @external would match sources and destinations outside the Incus network. However, my experiments are yielding unexpected results.
I’m testing this on a three-member Incus cluster with an OVN-type network:
root@incus-1:~# incus network ls | grep CREATED
| default | ovn | YES | 10.47.57.1/24 | none | | 4 | CREATED |
| uplink | physical | YES | | | | 1 | CREATED |
There are three container instances foo, bar and baz:
# incus ls
+------+---------+-------------------+------+-----------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+---------+-------------------+------+-----------+-----------+----------+
| bar | RUNNING | 10.47.57.3 (eth0) | | CONTAINER | 0 | incus-2 |
+------+---------+-------------------+------+-----------+-----------+----------+
| baz | RUNNING | 10.47.57.4 (eth0) | | CONTAINER | 0 | incus-3 |
+------+---------+-------------------+------+-----------+-----------+----------+
| foo | RUNNING | 10.47.57.2 (eth0) | | CONTAINER | 0 | incus-1 |
+------+---------+-------------------+------+-----------+-----------+----------+
Instances can reach internet and each other when no ACLs are involved:
root@foo:~# fping 10.47.57.3 10.47.57.4 8.8.8.8
10.47.57.3 is alive
10.47.57.4 is alive
8.8.8.8 is alive
When I add an ACL that should permit all egress traffic and only allow ingress traffic from internal addresses, I unexpectedly find that traffic between instances is blocked:
root@incus-1:~# incus network acl show default
name: default
description: ""
egress:
- action: allow
state: enabled
ingress:
- action: allow
source: '@internal'
state: enabled
config: {}
used_by:
- /1.0/networks/default
project: default
root@foo:~# fping 10.47.57.3 10.47.57.4 8.8.8.8
ICMP Host Unreachable from 10.47.57.3 for ICMP Echo sent to 10.47.57.3
ICMP Host Unreachable from 10.47.57.4 for ICMP Echo sent to 10.47.57.4
8.8.8.8 is alive
ICMP Host Unreachable from 10.47.57.3 for ICMP Echo sent to 10.47.57.3
ICMP Host Unreachable from 10.47.57.4 for ICMP Echo sent to 10.47.57.4
ICMP Host Unreachable from 10.47.57.3 for ICMP Echo sent to 10.47.57.3
ICMP Host Unreachable from 10.47.57.4 for ICMP Echo sent to 10.47.57.4
ICMP Host Unreachable from 10.47.57.3 for ICMP Echo sent to 10.47.57.3
ICMP Host Unreachable from 10.47.57.4 for ICMP Echo sent to 10.47.57.4
10.47.57.3 is unreachable
10.47.57.4 is unreachable
Interestingly, if I replace ‘@internal’ with the explicit subnet ‘10.47.57.0/24’, internal traffic is restored:
root@incus-1:~# incus network acl show default
name: default
description: ""
egress:
- action: allow
state: enabled
ingress:
- action: allow
source: 10.47.57.0/24
state: enabled
config: {}
used_by:
- /1.0/networks/default
project: default
root@foo:~# fping 10.47.57.3 10.47.57.4 8.8.8.8
10.47.57.3 is alive
10.47.57.4 is alive
8.8.8.8 is alive
Out of curiosity, I’ve also experimented with using selectors in egress rules. In some cases, the behavior matches my expectations:
root@incus-1:~# incus network acl show default
name: default
description: ""
egress:
- action: allow
destination: '@internal'
state: enabled
ingress:
- action: allow
source: 10.47.57.0/24
state: enabled
config: {}
used_by:
- /1.0/networks/default
project: default
root@foo:~# fping 10.47.57.3 10.47.57.4 8.8.8.8
10.47.57.3 is alive
10.47.57.4 is alive
ICMP Host Unreachable from 8.8.8.8 for ICMP Echo sent to 8.8.8.8
ICMP Host Unreachable from 8.8.8.8 for ICMP Echo sent to 8.8.8.8
ICMP Host Unreachable from 8.8.8.8 for ICMP Echo sent to 8.8.8.8
ICMP Host Unreachable from 8.8.8.8 for ICMP Echo sent to 8.8.8.8
8.8.8.8 is unreachable
Some times not:
root@incus-1:~# incus network acl show default
name: default
description: ""
egress:
- action: allow
destination: '@external'
state: enabled
ingress:
- action: allow
source: 10.47.57.0/24
state: enabled
config: {}
used_by:
- /1.0/networks/default
project: default
root@foo:~# fping 10.47.57.3 10.47.57.4 8.8.8.8
10.47.57.3 is alive
10.47.57.4 is alive
8.8.8.8 is alive
I am experimenting on Ubuntu 24.04 with latest updates and LTS Incus from Zabbly repo:
root@incus-1:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble
root@incus-1:~# incus version
Client version: 6.0.2
Server version: 6.0.2