Isolated Instances and Network Bridge ACLs

Incus
,
1

Hi Team,

There are quite a few conversations about networking, bridges, ACLs and how to isolate public services. In an effort to make this easier, I created this post: https://www.chuck-stack.org/ls/blog-incus-netbird-phoenixnap-isolated.html

Everything works except for one thing… The following ACL line is giving me an issue where DNS is not getting configured at cloud-init.

incus network acl rule add $INCUS_ACL egress action=reject destination=10.0.0.0/8

See full script below…

If I add the ACL to my profile after an instance is created, all works as expected.

If I run the script as is then create an instance, the instance cannot resolve dns. It gives an error: ping: google.com: Temporary failure in name resolution.

This tells me that the ACL is blocking something in the cloud-init that configures dns.

I need to fix this in one of two ways:

  • update my profile to hard-code the name servers
  • figure out what is getting blocked during cloud-init and unblock it in the below script

Both options are acceptable, and I have not been able to succeed with either resolution. Any ideas?

Full script:

# variables
INCUS_NETWORK=incusbr-iso
INCUS_PROFILE=isolated
INCUS_ACL=public-only

# Create artifacts
incus network create $INCUS_NETWORK
incus profile show default | incus profile create $INCUS_PROFILE
incus profile device set $INCUS_PROFILE eth0 network=$INCUS_NETWORK
incus network acl create $INCUS_ACL

# Get network interface details
MAIN_INTERFACE=$(ip route get 8.8.8.8 | grep -oP 'dev \K\S+')
echo MAIN_INTERFACE=$MAIN_INTERFACE
INCUS_ISO_SUBNET=$(ip -4 addr show $INCUS_NETWORK | grep -oP '(?<=inet\s)\d+(\.\d+){3}/\d+')
echo INCUS_ISO_SUBNET=$INCUS_ISO_SUBNET
INCUS_ISO_SUBNET_V6=$(ip -6 addr show $INCUS_NETWORK | grep -oP '(?<=inet6\s)[0-9a-f:]+/\d+' | grep -v '^fe80')
echo INCUS_ISO_SUBNET_V6=$INCUS_ISO_SUBNET_V6
INCUS_ISO_DNS=${INCUS_ISO_SUBNET%/*}
echo INCUS_ISO_DNS=$INCUS_ISO_DNS

# Create and configure an incus acl and apply it to our isolated network bridge
incus network acl rule add $INCUS_ACL ingress action=allow
incus network acl rule add $INCUS_ACL egress action=allow
#incus network acl rule add $INCUS_ACL egress action=allow destination=$INCUS_ISO_DNS/32
#incus network acl rule add $INCUS_ACL egress action=allow destination=$INCUS_ISO_DNS/32 protocol=udp destination_port=53
#incus network acl rule add $INCUS_ACL egress action=allow destination=$INCUS_ISO_DNS/32 protocol=tcp destination_port=53

# Block private networks
incus network acl rule add $INCUS_ACL egress action=reject destination=10.0.0.0/8
incus network acl rule add $INCUS_ACL egress action=reject destination=192.168.0.0/16
incus network acl rule add $INCUS_ACL egress action=reject destination=172.16.0.0/12

incus network set $INCUS_NETWORK security.acls=$INCUS_ACL
incus profile device set $INCUS_PROFILE eth0 security.acls=$INCUS_ACL
incus profile show $INCUS_PROFILE

# Allow $INCUS_NETWORK interface
sudo iptables -A INPUT -i $INCUS_NETWORK -j ACCEPT
sudo iptables -A OUTPUT -o $INCUS_NETWORK -j ACCEPT
sudo ip6tables -A INPUT -i $INCUS_NETWORK -j ACCEPT
sudo ip6tables -A OUTPUT -o $INCUS_NETWORK -j ACCEPT

# NAT and forwarding rules
sudo iptables -t nat -A POSTROUTING -s $INCUS_ISO_SUBNET -o $MAIN_INTERFACE -j MASQUERADE
sudo iptables -A FORWARD -i $INCUS_NETWORK -o $MAIN_INTERFACE -j ACCEPT
sudo iptables -A FORWARD -i $MAIN_INTERFACE -o $INCUS_NETWORK -j ACCEPT
sudo ip6tables -t nat -A POSTROUTING -s $INCUS_ISO_SUBNET_V6 -o $MAIN_INTERFACE -j MASQUERADE
sudo ip6tables -A FORWARD -i $INCUS_NETWORK -o $MAIN_INTERFACE -j ACCEPT
sudo ip6tables -A FORWARD -i $MAIN_INTERFACE -o $INCUS_NETWORK -j ACCEPT

## lauch instances in this isolated bridge
#incus launch images:debian/12/cloud delme-debian-isolated-01 --profile isolated
## show acl
#incus network acl show public-only

Thanks! Chuck

2

Try setting dns.namservers to 1.1.1.1,1.0.0.1 on your network. That way the DNS server won’t be in the 10.0.0.0/8 range anymore.

1 Like
3

Though that being said, this feels like a bug as in OVN at least we always make sure that DHCP and DNS is possible regardless of ACLs.

So you may want to report this at GitHub · Where software is built so we can sort it out.

4

@stgraber - confirmed success with the following:

INCUS_NAME_SERVERS="8.8.8.8,8.8.4.4"
incus network set $INCUS_NETWORK dns.nameservers=$INCUS_NAME_SERVERS

Do you want me to simply copy the above ‘as is’ into the github ticket/issue?

Also, I did some more investigating per our conversation about nftables. Turns out (based on research) that my server does not even use iptables. Instead, it uses iptables-nft, which translates iptables commands to nftables rules.

Executing sudo nft list ruleset shows all the iptables commands as nftables (nft).

I am including these details in this thread since Incus ACL rules are expressed as nft, and we do not want to mix the concepts.

Thanks for all your help!!
Chuck

5

I’d just mention that when applying an ACL that rejects all RFC1918 subnets but otherwise allows all traffic, that you’re unable to resolve DNS.

That will be enough for us to track this down.

6

@stgraber ,

Done: ACL against Network Bridge Drop DNS by defualt · Issue #1919 · lxc/incus · GitHub

Also, I rewrote everything to only use nftables. It is greatly simplified - thank you for the discussion!.

Here is what I believe:

  • the below does not create more permissive rules for either Netbird or Incus
  • I can delete the chuck-stack table without clearing any Incus details

Let me know if you have questions or concerns.

Here is the result:

#!/usr/sbin/nft -f

# This configuration assumes you have already 1) installed Netbird and 2) installed and configured Incus
# This configuration blocks all incoming traffic except through Netbird
#   - said another way, keep the server private and force entry through Netbird
table inet chuck-stack {
    chain input {
        type filter hook input priority 0; policy accept;

        # Allow established connections
        ct state related,established accept

        # Allow loopback
        iifname "lo" accept

        # Drop traffic from bond0 and its VLANs (public interfaces)
        iifname "bond0*" drop

        # Drop traffic from physical interfaces
        iifname "enp1s0f*" drop

        # Drop traffic from incusbr-iso* to incusbr* because incusbr-iso* bridges are considered isolated
        iifname "incusbr-iso*" oifname "incusbr*" drop

        # Drop traffic from incusbr-iso* to wt0 because incusbr-iso* bridges are considered isolated
        iifname "incusbr-iso*" oifname "wt0" drop
    }
}

# Add any additional custom rules below as needed
# to show rules: sudo nft list ruleset
# to add: sudo nft -f chuck-stack.conf
# to drop: sudo nft delete table inet chuck-stack
# to show interfaces: ip link show