Network Isolation by Project on Single Server Incus Host

[chuckboecking]

Hi Team,

I have a single incus host server (not clustered). I want to be able to isolate instances (from a networking perspective) to only communicate with containers in the same project and still be able to communicate with the outside world. My assumption is that you can create a bridged network per project; however, I do not know the best way to prevent communication between network bridges.

This instance will most likely have 10+ projects. If the host firewall needs to control communication, the networking commands need to grow linearly with each new project created (said another way, I do not want to prevent traffic with every other existing bridged network).

I also believe that incus firewall management can interfere with any iptable rules that I create (and visa versa).

Here is an example that seems to work. Is this a safe and complete solution assuming all bridge names begin with: incusbr (where using + as a wildcard)

#block traffic
sudo iptables -A FORWARD -i incusbr+ -o incusbr+ -j REJECT

#allow for exceptions
sudo iptables -A FORWARD -i incusbr4 -o incusbr8 -j ACCEPT

I know that you can create complete isolation using the following; however, I believe this leaves me using a tailscale type solution to support networking. Is this correct?
security.ipv4_filtering=true (prevents IPv4 spoofing)
security.ipv6_filtering=true (prevents IPv6 spoofing)
security.mac_filtering=true (prevents MAC spoofing)
security.port_isolation=true (prevents container to container traffic)

Any advice?
Thanks in advance - Chuck

[stgraber (Stéphane Graber)]

You should be able to create an ACL and assign it to your bridge within Incus, you’d like put reject rules for the usual RFC1918 space so your instances can access the public internet but not anything private.

[chuckboecking]

Hi @stgraber , thank you for your quick reply! I would like a little more help if you are willing… Here are my thoughts:

• I have searched the entire internet, and I cannot find a single example of an “incus network acl rule add …” command so that I can see what the [properties…] might look like and what my options are.
• If I understood you correctly, I would need to specify (or programmatically find) the subnet IP space for both v4 and v6 for the newly created bridge so that I could create the appropriate acl deny statements.
• Would you kindly create example ‘acl create …’ and ‘acl add …’ statements so that I may better understand your recommendation? I am happy to sponsor your efforts.

Let’s assume a newly created incus instance with incusbr0.

Let’s assume I created a new bridge network using: incus network create incusbr1

The following is what was created: incus network show incusbr1

config:
  ipv4.address: 10.52.28.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:903d:e8fa:756e::1/64
  ipv6.nat: "true"
description: ""
name: incusbr1
type: bridge
used_by:
- /1.0/instances/delme-03-br1
managed: true
status: Created
locations:
- none

I can create the network acl using: incus network acl create incusbr1-acl-01

I have no idea how to finish this statement to add a rule: incus network acl rule add incusbr1-acl-01 ingress [properties…]

Once the rules are added, you can set the acl to the new bridge network: incus network set incusbr1 security.acls=“incusbr1-acl-01”

Thank you for your time and consideration! Chuck

[stgraber (Stéphane Graber)]

root@v1:~# incus project create foo
Project foo created
root@v1:~# incus project create bar
Project bar created

root@v1:~# incus profile show default | incus profile edit default --project foo
root@v1:~# incus profile show default | incus profile edit default --project bar

root@v1:~# incus network create br-foo
Network br-foo created
root@v1:~# incus network create br-bar
Network br-bar created

root@v1:~# incus profile device set default eth0 network=br-foo --project foo
root@v1:~# incus profile device set default eth0 network=br-bar --project bar

root@v1:~# incus launch images:ubuntu/22.04 u1 --project foo
Launching u1
root@v1:~# incus launch images:ubuntu/22.04 u2 --project bar
Launching u2
root@v1:~# incus list --all-projects
+---------+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| PROJECT | NAME |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+---------+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| bar     | u2   | RUNNING | 10.225.141.73 (eth0) | fd42:e575:3737:756a:216:3eff:fe79:b3b6 (eth0) | CONTAINER | 0         |
+---------+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| foo     | u1   | RUNNING | 10.190.44.137 (eth0) | fd42:df65:6f8e:f6a5:216:3eff:fe34:6f0d (eth0) | CONTAINER | 0         |
+---------+------+---------+----------------------+-----------------------------------------------+-----------+-----------+

root@v1:~# incus exec u1 bash --project foo
root@u1:~# ping 10.225.141.73
PING 10.225.141.73 (10.225.141.73) 56(84) bytes of data.
64 bytes from 10.225.141.73: icmp_seq=1 ttl=63 time=0.079 ms
^C
--- 10.225.141.73 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.079/0.079/0.079/0.000 ms
root@u1:~# 
exit

root@v1:~# incus network acl create public-only
Network ACL public-only created
root@v1:~# incus network acl rule add public-only ingress action=allow
root@v1:~# incus network acl rule add public-only egress action=allow
root@v1:~# incus network acl rule add public-only egress action=reject destination=10.0.0.0/8
root@v1:~# incus network acl rule add public-only egress action=reject destination=192.168.0.0/16
root@v1:~# incus network acl rule add public-only egress action=reject destination=172.16.0.0/12

root@v1:~# incus network set br-bar security.acls=public-only
root@v1:~# incus network set br-foo security.acls=public-only

root@v1:~# incus exec u1 bash --project foo
root@u1:~# ping 10.225.141.73
PING 10.225.141.73 (10.225.141.73) 56(84) bytes of data.
From 10.190.44.1 icmp_seq=1 Destination Port Unreachable
^C
--- 10.225.141.73 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

root@u1:~# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=8.91 ms
^C
--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 8.911/8.911/8.911/0.000 ms
root@u1:~# 
exit
root@v1:~# 

[chuckboecking]

Hi @stgraber - thank you for your kind help with this thread!!

[Andrew_Wilson]

I would like to add a comment to help remind folks (like me) to set their firewall to allow for traffic to/from the new network(s), otherwise, like me, you might scratch around wondering why you can’t get connectivity. It took me ten minutes to figure it out (again) as I launched a container and it could not get an IP address from incus, and nor did it have connectivity. As usual, the error was my fault again:

E.g. for anyone using the uncomplicated firewall, ufw, I use these commands to get the firewall to talk nicely to the new incus ‘foo’ network per the above:

sudo ufw allow in on foo
sudo ufw route allow in on foo
sudo ufw route allow out on foo

Repeat for network bar too. And if you use iptables or some other application, hopefully these instructions still point you on how to do this.

A dated but still useful post (albeit for lxd) explains this well:

Lxd bridge doesn't work with IPv4 and UFW with nftables

As you can see getting the various firewall implementations to play nicely together is a non-trivial task. In the meantime these commands seems to suffice to allow traffic from lxdbr0 interface to the LXD host and for traffic from lxdbr0 to be routed to the external network without allowing all external inbound traffic: sudo ufw allow in on lxdbr0 sudo ufw route allow in on lxdbr0 sudo ufw route allow out on lxdbr0 See our page in the docs about this also: