If I add the ACL to my profile after an instance is created, all works as expected.
If I run the script as is then create an instance, the instance cannot resolve dns. It gives an error: ping: google.com: Temporary failure in name resolution.
This tells me that the ACL is blocking something in the cloud-init that configures dns.
I need to fix this in one of two ways:
update my profile to hard-code the name servers
figure out what is getting blocked during cloud-init and unblock it in the below script
Both options are acceptable, and I have not been able to succeed with either resolution. Any ideas?
INCUS_NAME_SERVERS="8.8.8.8,8.8.4.4"
incus network set $INCUS_NETWORK dns.nameservers=$INCUS_NAME_SERVERS
Do you want me to simply copy the above ‘as is’ into the github ticket/issue?
Also, I did some more investigating per our conversation about nftables. Turns out (based on research) that my server does not even use iptables. Instead, it uses iptables-nft, which translates iptables commands to nftables rules.
Executing sudo nft list ruleset shows all the iptables commands as nftables (nft).
I am including these details in this thread since Incus ACL rules are expressed as nft, and we do not want to mix the concepts.
Also, I rewrote everything to only use nftables. It is greatly simplified - thank you for the discussion!.
Here is what I believe:
the below does not create more permissive rules for either Netbird or Incus
I can delete the chuck-stack table without clearing any Incus details
Let me know if you have questions or concerns.
Here is the result:
#!/usr/sbin/nft -f
# This configuration assumes you have already 1) installed Netbird and 2) installed and configured Incus
# This configuration blocks all incoming traffic except through Netbird
# - said another way, keep the server private and force entry through Netbird
table inet chuck-stack {
chain input {
type filter hook input priority 0; policy accept;
# Allow established connections
ct state related,established accept
# Allow loopback
iifname "lo" accept
# Drop traffic from bond0 and its VLANs (public interfaces)
iifname "bond0*" drop
# Drop traffic from physical interfaces
iifname "enp1s0f*" drop
# Drop traffic from incusbr-iso* to incusbr* because incusbr-iso* bridges are considered isolated
iifname "incusbr-iso*" oifname "incusbr*" drop
# Drop traffic from incusbr-iso* to wt0 because incusbr-iso* bridges are considered isolated
iifname "incusbr-iso*" oifname "wt0" drop
}
}
# Add any additional custom rules below as needed
# to show rules: sudo nft list ruleset
# to add: sudo nft -f chuck-stack.conf
# to drop: sudo nft delete table inet chuck-stack
# to show interfaces: ip link show
With Incus 6.20 and IncusOS 202512250102 the DNS problem seems to be back for DNS. When having an ACL that blocks access to its own subnet (e.g. 192.168.3.0/24), getting DHCP lease from the router (in this case 192.168.3.1) is possible but not DNS. security.acls.default.ingress.action=allow and security.acls.default.egress.action=allow are set to allow.
I can only speak for non OVN interfaces that are setup like:
Additionally, when setting security.acls.default.ingress.action=reject and security.acls.default.egress.action=allow, without any additional rules, there is no connectivity at all. No DHCP Addresses are assigned and also DNS is not working. Which to my understanding should work if DHCP and DNS are excluded form ACLs,
Hmm, so you’re using OVN as just a virtual L2 which is bridging everything to a physical VLAN through one of the servers?
ACLs are definitely going to struggle because of that. Our generated ACLS assume that OVN knows the DHCP address, DNS address, … those don’t exist in this environment so I wouldn’t expect there to be any rules allowing such traffic.
It also wouldn’t be safe to just outright allow all DNS or DHCP traffic as you could have a malicious instance on that network run a DHCP server, race the real one and then act as de-facto router and DNS, intercepting all traffic.
When OVN is just used to stretch a physical L2 network in that way, I think I’d assume that ACLs would effectively work purely statelessly and with no default rules as I don’t see how OVN could implement anything more complex than that without it being in charge of addressing and routing.
Ah, that explains a lot. It was not clear to me, by only reading the docs. I thought it would also be stateful on bridge networks without OVN.
We have all routing and stuff done on our external firewall that uses VLANs for network segmentation. It also provides DHCP and DNS. For public internet facing sevices, our goal is to also separate the virtual instances of the DMZ VLAN to no being able to talk to each other.
What I could successfully do is the following. Maybe it helps others trying to do the same.
And then assign it to Containers / VMs that are in the DMZ.
@stgraber, is this “safe” to use or would the security.acls.default.egress.action: allow and security.acls.default.ingress.action: allow open up unintended traffic to other VLAN enabled bridges on the same server or to Incus networks created with --type=physical vlan=10 and work around the rules on our dedicated firewall? I guess not, as the Incus ACLs only “live” on the brindge and do not enable any unwanted routing capabilities on Incus, right? Is there another way to accomplish the DMZ virutal instances not talking to each other on Incus level?