Isolated Instances and Network Bridge ACLs

Hi Team,

There are quite a few conversations about networking, bridges, ACLs and how to isolate public services. In an effort to make this easier, I created this post: https://www.chuck-stack.org/ls/blog-incus-netbird-phoenixnap-isolated.html

Everything works except for one thing… The following ACL line is giving me an issue where DNS is not getting configured at cloud-init.

incus network acl rule add $INCUS_ACL egress action=reject destination=10.0.0.0/8

See full script below…

If I add the ACL to my profile after an instance is created, all works as expected.

If I run the script as is then create an instance, the instance cannot resolve dns. It gives an error: ping: google.com: Temporary failure in name resolution.

This tells me that the ACL is blocking something in the cloud-init that configures dns.

I need to fix this in one of two ways:

  • update my profile to hard-code the name servers
  • figure out what is getting blocked during cloud-init and unblock it in the below script

Both options are acceptable, and I have not been able to succeed with either resolution. Any ideas?

Full script:

# variables
INCUS_NETWORK=incusbr-iso
INCUS_PROFILE=isolated
INCUS_ACL=public-only

# Create artifacts
incus network create $INCUS_NETWORK
incus profile show default | incus profile create $INCUS_PROFILE
incus profile device set $INCUS_PROFILE eth0 network=$INCUS_NETWORK
incus network acl create $INCUS_ACL

# Get network interface details
MAIN_INTERFACE=$(ip route get 8.8.8.8 | grep -oP 'dev \K\S+')
echo MAIN_INTERFACE=$MAIN_INTERFACE
INCUS_ISO_SUBNET=$(ip -4 addr show $INCUS_NETWORK | grep -oP '(?<=inet\s)\d+(\.\d+){3}/\d+')
echo INCUS_ISO_SUBNET=$INCUS_ISO_SUBNET
INCUS_ISO_SUBNET_V6=$(ip -6 addr show $INCUS_NETWORK | grep -oP '(?<=inet6\s)[0-9a-f:]+/\d+' | grep -v '^fe80')
echo INCUS_ISO_SUBNET_V6=$INCUS_ISO_SUBNET_V6
INCUS_ISO_DNS=${INCUS_ISO_SUBNET%/*}
echo INCUS_ISO_DNS=$INCUS_ISO_DNS

# Create and configure an incus acl and apply it to our isolated network bridge
incus network acl rule add $INCUS_ACL ingress action=allow
incus network acl rule add $INCUS_ACL egress action=allow
#incus network acl rule add $INCUS_ACL egress action=allow destination=$INCUS_ISO_DNS/32
#incus network acl rule add $INCUS_ACL egress action=allow destination=$INCUS_ISO_DNS/32 protocol=udp destination_port=53
#incus network acl rule add $INCUS_ACL egress action=allow destination=$INCUS_ISO_DNS/32 protocol=tcp destination_port=53

# Block private networks
incus network acl rule add $INCUS_ACL egress action=reject destination=10.0.0.0/8
incus network acl rule add $INCUS_ACL egress action=reject destination=192.168.0.0/16
incus network acl rule add $INCUS_ACL egress action=reject destination=172.16.0.0/12

incus network set $INCUS_NETWORK security.acls=$INCUS_ACL
incus profile device set $INCUS_PROFILE eth0 security.acls=$INCUS_ACL
incus profile show $INCUS_PROFILE

# Allow $INCUS_NETWORK interface
sudo iptables -A INPUT -i $INCUS_NETWORK -j ACCEPT
sudo iptables -A OUTPUT -o $INCUS_NETWORK -j ACCEPT
sudo ip6tables -A INPUT -i $INCUS_NETWORK -j ACCEPT
sudo ip6tables -A OUTPUT -o $INCUS_NETWORK -j ACCEPT

# NAT and forwarding rules
sudo iptables -t nat -A POSTROUTING -s $INCUS_ISO_SUBNET -o $MAIN_INTERFACE -j MASQUERADE
sudo iptables -A FORWARD -i $INCUS_NETWORK -o $MAIN_INTERFACE -j ACCEPT
sudo iptables -A FORWARD -i $MAIN_INTERFACE -o $INCUS_NETWORK -j ACCEPT
sudo ip6tables -t nat -A POSTROUTING -s $INCUS_ISO_SUBNET_V6 -o $MAIN_INTERFACE -j MASQUERADE
sudo ip6tables -A FORWARD -i $INCUS_NETWORK -o $MAIN_INTERFACE -j ACCEPT
sudo ip6tables -A FORWARD -i $MAIN_INTERFACE -o $INCUS_NETWORK -j ACCEPT

## lauch instances in this isolated bridge
#incus launch images:debian/12/cloud delme-debian-isolated-01 --profile isolated
## show acl
#incus network acl show public-only

Thanks! Chuck

Try setting dns.namservers to 1.1.1.1,1.0.0.1 on your network. That way the DNS server won’t be in the 10.0.0.0/8 range anymore.

Though that being said, this feels like a bug as in OVN at least we always make sure that DHCP and DNS is possible regardless of ACLs.

So you may want to report this at GitHub · Where software is built so we can sort it out.

@stgraber - confirmed success with the following:

INCUS_NAME_SERVERS="8.8.8.8,8.8.4.4"
incus network set $INCUS_NETWORK dns.nameservers=$INCUS_NAME_SERVERS

Do you want me to simply copy the above ‘as is’ into the github ticket/issue?

Also, I did some more investigating per our conversation about nftables. Turns out (based on research) that my server does not even use iptables. Instead, it uses iptables-nft, which translates iptables commands to nftables rules.

Executing sudo nft list ruleset shows all the iptables commands as nftables (nft).

I am including these details in this thread since Incus ACL rules are expressed as nft, and we do not want to mix the concepts.

Thanks for all your help!!
Chuck

I’d just mention that when applying an ACL that rejects all RFC1918 subnets but otherwise allows all traffic, that you’re unable to resolve DNS.

That will be enough for us to track this down.

@stgraber ,

Done: ACL against Network Bridge Drop DNS by defualt · Issue #1919 · lxc/incus · GitHub

Also, I rewrote everything to only use nftables. It is greatly simplified - thank you for the discussion!.

Here is what I believe:

  • the below does not create more permissive rules for either Netbird or Incus
  • I can delete the chuck-stack table without clearing any Incus details

Let me know if you have questions or concerns.

Here is the result:

#!/usr/sbin/nft -f

# This configuration assumes you have already 1) installed Netbird and 2) installed and configured Incus
# This configuration blocks all incoming traffic except through Netbird
#   - said another way, keep the server private and force entry through Netbird
table inet chuck-stack {
    chain input {
        type filter hook input priority 0; policy accept;

        # Allow established connections
        ct state related,established accept

        # Allow loopback
        iifname "lo" accept

        # Drop traffic from bond0 and its VLANs (public interfaces)
        iifname "bond0*" drop

        # Drop traffic from physical interfaces
        iifname "enp1s0f*" drop

        # Drop traffic from incusbr-iso* to incusbr* because incusbr-iso* bridges are considered isolated
        iifname "incusbr-iso*" oifname "incusbr*" drop

        # Drop traffic from incusbr-iso* to wt0 because incusbr-iso* bridges are considered isolated
        iifname "incusbr-iso*" oifname "wt0" drop
    }
}

# Add any additional custom rules below as needed
# to show rules: sudo nft list ruleset
# to add: sudo nft -f chuck-stack.conf
# to drop: sudo nft delete table inet chuck-stack
# to show interfaces: ip link show

With Incus 6.20 and IncusOS 202512250102 the DNS problem seems to be back for DNS. When having an ACL that blocks access to its own subnet (e.g. 192.168.3.0/24), getting DHCP lease from the router (in this case 192.168.3.1) is possible but not DNS. security.acls.default.ingress.action=allow and security.acls.default.egress.action=allow are set to allow.

I can only speak for non OVN interfaces that are setup like:

incus network create incusbr1 --type=bridge \
bridge.external_interfaces=<interface_name>/_pMAC/VLANID \
ipv4.address=none \
ipv6.address=none 

Additionally, when setting security.acls.default.ingress.action=reject and security.acls.default.egress.action=allow, without any additional rules, there is no connectivity at all. No DHCP Addresses are assigned and also DNS is not working. Which to my understanding should work if DHCP and DNS are excluded form ACLs,

Hmm, so you’re using OVN as just a virtual L2 which is bridging everything to a physical VLAN through one of the servers?

ACLs are definitely going to struggle because of that. Our generated ACLS assume that OVN knows the DHCP address, DNS address, … those don’t exist in this environment so I wouldn’t expect there to be any rules allowing such traffic.

It also wouldn’t be safe to just outright allow all DNS or DHCP traffic as you could have a malicious instance on that network run a DHCP server, race the real one and then act as de-facto router and DNS, intercepting all traffic.

When OVN is just used to stretch a physical L2 network in that way, I think I’d assume that ACLs would effectively work purely statelessly and with no default rules as I don’t see how OVN could implement anything more complex than that without it being in charge of addressing and routing.

Ah, that explains a lot. It was not clear to me, by only reading the docs. I thought it would also be stateful on bridge networks without OVN.

We have all routing and stuff done on our external firewall that uses VLANs for network segmentation. It also provides DHCP and DNS. For public internet facing sevices, our goal is to also separate the virtual instances of the DMZ VLAN to no being able to talk to each other.

What I could successfully do is the following. Maybe it helps others trying to do the same.

On InucsOS incus admin os system network edit

config:
[...]
  - hwaddr: AB:AB:AB:AB:AB
    name: eno1
    lldp: false
    required_for_online: "no"
    roles:
    - instances
    vlan_tags:
    - 10
    - 20 # DMZ
    - 30

Then on Incus

incus network create incusbr20 --type=bridge \
bridge.external_interfaces=vlan20/_pABABABABAB/20 \
ipv4.address=none \
ipv6.address=none 

Then set up an ACL:

name: DMZ-ACL
description: ''
egress:
  - action: reject
    destination: 192.168.20.2-255 # DMZ
    state: enabled
  - action: reject
    destination: 2a02:123::2-2a02:123::ffff:ffff:ffff:ffff# DMZ
    state: enabled
ingress: []
config: {}

where 192.168.20.1 and 2a02:123::1 are the Gatway of the firewall for VLAN 20 (DMZ) providing DHCP and DNS.

Then add a profile to Incus

name: "DMZ20_no-host-to-host''
devices:
  eth0:
    network: incusbr20
    security.acls: DMZ-ACL
    security.acls.default.egress.action: allow
    security.acls.default.ingress.action: allow
    type: nic
config: {}
project: default

And then assign it to Containers / VMs that are in the DMZ.

@stgraber, is this “safe” to use or would the security.acls.default.egress.action: allow and security.acls.default.ingress.action: allow open up unintended traffic to other VLAN enabled bridges on the same server or to Incus networks created with --type=physical vlan=10 and work around the rules on our dedicated firewall? I guess not, as the Incus ACLs only “live” on the brindge and do not enable any unwanted routing capabilities on Incus, right? Is there another way to accomplish the DMZ virutal instances not talking to each other on Incus level?