LXD stopped generating firewall rules after switch to core20

LXD
,
1

Interesting case: after reboot. LXD no longer generates firewall rules. It seems to silently fail.

Ubuntu 20.04 LTS with latest LXD from snap.

UFW successfully starts and generates its default rules.

We have many fairly identical hosts that all work down to the UFW rules, so I suspect the cause is some incident on this specific host. But everything else about LXD working makes this very suspicious.

LXD log, LXD monitor, starting LXD in foreground with debug mode, syslog, systemd tools have come up with only one discrepancy from the other hosts. The failed host shows a warning about br_netfilter that other do not (it is not loaded on the others):

Jun 17 17:21:08 h2 lxd.daemon[8913]: t=2021-06-17T17:21:08+0000 lvl=warn msg="Proxy bridge netfilter not enabled: br_netfilter not loaded: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory. Instances using the bridge will not be able to connect to the proxy's listen IP"

Adding the module to /etc/modules, updating initramfs and rebooting makes the warning go away but does not fix the issue.

Flipping ipv4/6.firewall on and off has no effect and does not produce any result, monitor shows:

location: none
metadata:
  context:
    driver: bridge
    network: lxdbr0
    project: default
  level: dbug
  message: Clearing firewall
timestamp: "2021-06-17T18:48:20.437222175Z"
type: logging


location: none
metadata:
  context:
    driver: bridge
    network: lxdbr0
    project: default
  level: dbug
  message: Setting up firewall
timestamp: "2021-06-17T18:48:21.127576861Z"
type: logging

Disabling UFW and rebooting has no effect on LXD.

It seems as if LXD on this host simply and silently stopped interacting with iptables.

Any thoughts on where to look for more details or to debug this?

2

Can you show lxc info | grep firewall: ?

3

A very curious side is that some (all?) containers still respond according to their proxy rules if turning off UFW at least.

But nftables is not installed and /etc/alternatives/ip* point at iptables-legacy

The log says it loaded nftables, which is no surprise since it should manage iptables
t=2021-06-17T18:29:42+0000 lvl=info msg=“Firewall loaded driver “nftables””

4

$ lxc info | grep firewall:
firewall: nftables

5

Ok, so it would make sense that you’re not seeing the rules in xtables then if they’re in nftables.

The two may however end up conflicting and LXD should have detected the pre-existing xtables rules…

6

The other hosts also show nftables as the firewall used but their rules show up in iptables-save as usual. I think xtables is only used on old kernels?

7

I have an idea: is it possible that the firewall selection code in LXD thought that iptables is not available if it was busy? Our fail2ban rules are quite long, and it tends to block iptables after a reboot.

8

There’s definitely an issue with the core20 based snap not detecting prior xtables entries and incorrectly moving forward with nft I think. I’m looking into this now.

Jun 17 19:54:45 vm02 lxd.daemon[3591]: t=2021-06-17T19:54:45+0000 lvl=dbug msg="Firewall detected \"xtables\" incompatibility: Backend command \"ebtables\" is an nftables shim"
9

I pushed a tentative fix to latest/edge just now, if this builds properly and behaves here, I’ll send it to latest/stable along with some other fixes for issues we’ve seen reported today.

Basically the issue is that core20 for some weird reason has iptables/ip6tables using the xtables backend but has arptables/ebtables use the nft backend… This confuses LXD and so has it fallback to nft-only which can then cause some confusion with existing firewalls…

@tomp this likely can explain a few other of the issues we’ve seen

1 Like
10

Oh yes indeed that would explain some of the other effects people have been seeing after fixing their dnsmasq config.

11

This is a workaround in meantime Lxd bridge doesn't work with IPv4 and UFW with nftables - #17 by tomp

1 Like
12

I can confirm that the current set of packages as of this writing work:

Name    Version   Rev    Tracking       Publisher   Notes
core18  20210507  2066   latest/stable  canonicalâś“  base
core20  20210429  1026   latest/stable  canonicalâś“  base
lxd     4.15      20789  latest/stable  canonicalâś“  -
snapd   2.51      12159  latest/stable  canonicalâś“  snapd

For anyone looking for the same, you have to restart lxd, i.e.:

sudo snap refresh
sudo snap restart lxd
1 Like