I’m using incus 7.0, and the requirement I have is very simple: I want to find the cleanest way to disable NAT for communication between two different incus managed bridges on the same host.
Specific example: I have incusbr0 on 100.125.0.1/22 and wifi0 on 100.126.0.1/22. A client on wifi0 which pings a container on incusbr0 is able to get through, but the source address seen by the destination container is always 100.125.0.1, because it has been source-NAT’d.
Incus adds nft rules of this form:
table inet incus {
chain pstrt.incusbr0 {
type nat hook postrouting priority 100; policy accept;
ip saddr 100.125.0.0/22 ip daddr != 100.125.0.0/22 masquerade
...
chain pstrt.wifi0 {
type nat hook postrouting priority 100; policy accept;
ip saddr 100.126.0.0/22 ip daddr != 100.126.0.0/22 masquerade
Therefore, any traffic leaving wifi0 gets masqueraded, even if its destination is on incusbr0.
I’m raising this as a discussion because there are several possible solutions, and I would like to get some consensus on the approach before raising a more specific feature request.
What I tried at first was to create my own NAT rule independent of the incus rules, trying to catch all traffic with destination 100.64.0.0/10 (covering both networks) and set “accept” rather than “masquerade” - using iptables for ease of testing, but this was mapped into nft:
table ip nat {
chain PREROUTING {
type nat hook prerouting priority -100; policy accept;
}
chain POSTROUTING {
type nat hook postrouting priority 100; policy accept;
ip daddr 100.64.0.0/10 counter packets 91 bytes 34140 accept
What I found was:
- The postrouting rules for iptables and incus are both priority 100, so you don’t know which is executed first. That could easily be fixed if I created my own chain directly in nft and selected the priority.
- Here’s the kicker though. In nft, an “accept” action finishes the current chain; but if there are other chains on the same hook with following priority, they still get run. In other words, the “accept” action is not final.
I found this via stackexchange, and here’s what the nftables wiki says:
NOTE: If a packet is accepted and there is another chain, bearing the same hook type and with a later priority, then the packet will subsequently traverse this other chain. Hence, an accept verdict - be it by way of a rule or the default chain policy - isn’t necessarily final. However, the same is not true of packets that are subjected to a drop verdict. Instead, drops take immediate effect, with no further rules or chains being evaluated.
Maybe that makes sense for packet filtering, but it’s a real pain for NAT where AFAICS you can’t just say “don’t NAT, and that’s the final verdict”.
So far, the only solution I’ve been able to find without hacking incus is:
- Disable incus NAT entirely (i.e. set
ipv4.nat=falseon all incus-managed networks) - Add a new rule directly on the host itself, which does the required NAT. For example, using
/etc/ufw/before.rulesI can add a rule for the covering block:
# Setting up NAT
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -s 100.64.0.0/10 ! -d 100.64.0.0/10 -j MASQUERADE
# don't delete the 'COMMIT' line or these rules won't be processed
COMMIT
But that’s pretty darned ugly.
Given the way that NAT rules interact, I think the only way to have incus manage this is if it adds custom rules to its own chains (i.e. chain pstrt.<network>). But even then it would be very hard, because incus itself creates multiple chains (one per managed network), and I can’t see how you can set a “don’t NAT” action in one chain which isn’t overridden by another chain.
One option would be to make the NAT rules list all the incus managed networks as “do not NAT” destinations: e.g. (untested)
table inet incus {
chain pstrt.incusbr0 {
type nat hook postrouting priority 100; policy accept;
ip saddr 100.125.0.0/22 ip daddr != 100.125.0.0/22 ip daddr != 100.126.0.0/22 masquerade
Or better, use nft address sets; otherwise it’s not going to scale well if you have hundreds of networks. If incus were to create a named address set, and list all the incus-managed networks, the same network set could be used on all chains, and then all internal-to-internal traffic is not NAT’d.
Another way is if the user could nominate one or more aggregate netblocks to exclude NAT. For example:
incus network set incusbr0 ipv4.nat.exclude_destinations="100.64.0.0/10 192.168.0.0/16"
which might change the NAT rule to be:
table inet incus {
chain pstrt.incusbr0 {
type nat hook postrouting priority 100; policy accept;
ip saddr 100.125.0.0/22 ip daddr != 100.64.0.0/10 ip daddr != 192.168.0.0/16 masquerade
If it’s not a global setting then you’d still have to remember to set this on every incus managed network, although the default (if not explicitly configured) could be an address set containing all incus managed networks.
Other options include explicitly listing the interfaces that you want to NAT through, but that’s a pain if outbound interface towards the Internet changes. Or listing all the interfaces that you don’t want to NAT through (i.e. all the incus interfaces), but that could become a very long list.
Any suggestions? Is there some obvious option I’ve missed here?
Many thanks,
Brian.
P.S. while going down this rabbit hole, I spent some time experimenting with ipv4.nat.order={before,after} - only to find this is explicitly ignored by the Nftables firewall driver in incus. I’ve raised an issue to improve the documentation