I was just about to create a new topic with the same questions as @m1cha but gladly I found this thread and I don’t need to explain the whole issue with lxd’s nft tables.
But it bugs me slightly to see @tomp 's statement.
Hopefully in the future we will see some consensus around how applications should coexist when using nftables, or perhaps its behavior will be changed to allow an allow rule to apply irrespective of a drop rule in another table.
Although it’s an understandable issue on nftables, this comment seems to indicate there will be no solution unless someone else solves the problem. This will leave lxd in an awkward situation as nftables usage increases.
Expected behavior
It should be possible to obtain a hardened instance firewall without crippling LXD. It should be possible to route forward only packets to LXD managed networks while denying everything else. It should also be possible to harden input firewall rules to accept only a few rules and to be able to add a few extra LXD generated rules.
How it works currently
To allow for instance internet access, you need to allow all forward packets unless you configure firewall rules by hand. Similar issues with input and output rules.
The issue
It seems the main issue is that lxd is using a base chain instead of a regular chain. Indeed the base chain is useless if there is another base chain using the same hook which rejects packets, as was the case in this thread, for example.
A secondary base chain would only make sense to add drop veredict statements. As is, a secondary base chain with only accept rules is pointless.
A possible solution
Be able to change the behaviour of LXD firewall generated rules so that LXD rules are contained in a regular chain and are inserted as a jump from another base chain set of rules.
Just as there are network configuration options which control a few aspects of how these rules are added, how about adding a bit more so the user can configure them correctly?
Make the defaults to add rules as they are so there is no change for anyone who doesn’t run into these issues. As @m1cha was mentioning, add a ipv4.nft.forward.policy configuration option so users can change the default policy of these rules.
It would also be nice to be able to add the chains as regular and not base chains and be able to add something like a vmap to jump from another base chain into the correct lxd regular chain.
Something like ip4.nft.input.type=jump, ip4.nft.input.table=filter, ip4.nft.input.chain=INPUT, ip4.nft.input.vmap_line=5 would produce a regular chain with the input rules and it would add a jump statement to the INPUT chain in the filter table on line 5. Here I’m guessing ip4.nft.input.type could be either base, jump or goto in which case both jump and goto would make a custom input chain and the vmap would either jump or goto the correct table in the specified line. Default value would be base.