Lxd bridge doesn't work with IPv4 and UFW with nftables

It’s changing to UP but no address attached to the container:

root@HOLODECK:~# lxc launch ubuntu:20.04 test
Creating test
Starting test                               

root@HOLODECK:~# lxc list
+------+---------+------+------+-----------+-----------+
| NAME |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+------+---------+------+------+-----------+-----------+
| test | RUNNING |      |      | CONTAINER | 0         |
+------+---------+------+------+-----------+-----------+

root@HOLODECK:~# ip addr show lxdbr0
8: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group     default qlen 1000
    link/ether 00:16:3e:07:49:7b brd ff:ff:ff:ff:ff:ff
    inet 10.52.135.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe07:497b/64 scope link 
       valid_lft forever preferred_lft forever

Then, when I jump into the container:

root@HOLODECK:~# lxc shell test
root@test:~# ip -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128 
eth0@if10        UP             fe80::216:3eff:feb3:15db/64 
root@test:~# ip addr show eth0@if10 
Device "eth0@if10" does not exist.
root@test:~# ip addr show eth0
9: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:b3:15:db brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::216:3eff:feb3:15db/64 scope link 
       valid_lft forever preferred_lft forever

There is no IPv4 addresses.

OK, so the interface is up though so thats good.

Next step are to resolve why DHCP isn’t working.

Take a look here to ensure its not the usual candidates preventing DHCP working:

@tomp Thank you for your help and expertise, I’m finally getting somewhere.

root@HOLODECK:~# sudo ss -ulpn | grep dnsmasq
UNCONN 0      0               10.52.135.1:53         0.0.0.0:*    users:(("dnsmasq",pid=7071,fd=6))         
UNCONN 0      0             192.168.122.1:53         0.0.0.0:*    users:(("dnsmasq",pid=2608,fd=5))         
UNCONN 0      0            0.0.0.0%lxdbr0:67         0.0.0.0:*    users:(("dnsmasq",pid=7071,fd=4))         
UNCONN 0      0            0.0.0.0%virbr0:67         0.0.0.0:*    users:(("dnsmasq",pid=2608,fd=3))

root@HOLODECK:~# ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

root@HOLODECK:~# sudo iptables-save | grep "dport 53"
-A ufw-before-input -d 224.0.0.251/32 -p udp -m udp --dport 5353 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 53 -j ACCEPT

So there is no firewall rules to allow DNS on LXD bridge. Should they be added automatically on initialization? I’m using LXD with default deny firewalls on other hosts (Focal instead of Groovy like this one) and I can confirm that appropriate rules are generated automatically at some point.

Show output of sudo iptables-save and sudo nft list ruleset please

Sure.

root@HOLODECK:~# iptables-save 
# Generated by iptables-save v1.8.5 on Mon Jan 25 15:50:52 2021
*filter
:INPUT DROP [11:352]
:FORWARD DROP [6:374]
:OUTPUT ACCEPT [0:0]
:ufw-before-logging-input - [0:0]
:ufw-before-logging-output - [0:0]
:ufw-before-logging-forward - [0:0]
:ufw-before-input - [0:0]
:ufw-before-output - [0:0]
:ufw-before-forward - [0:0]
:ufw-after-input - [0:0]
:ufw-after-output - [0:0]
:ufw-after-forward - [0:0]
:ufw-after-logging-input - [0:0]
:ufw-after-logging-output - [0:0]
:ufw-after-logging-forward - [0:0]
:ufw-reject-input - [0:0]
:ufw-reject-output - [0:0]
:ufw-reject-forward - [0:0]
:ufw-track-input - [0:0]
:ufw-track-output - [0:0]
:ufw-track-forward - [0:0]
:LIBVIRT_INP - [0:0]
:LIBVIRT_OUT - [0:0]
:LIBVIRT_FWO - [0:0]
:LIBVIRT_FWI - [0:0]
:LIBVIRT_FWX - [0:0]
:ufw-logging-deny - [0:0]
:ufw-logging-allow - [0:0]
:ufw-skip-to-policy-input - [0:0]
:ufw-skip-to-policy-output - [0:0]
:ufw-skip-to-policy-forward - [0:0]
:ufw-not-local - [0:0]
:ufw-user-input - [0:0]
:ufw-user-output - [0:0]
:ufw-user-forward - [0:0]
:ufw-user-logging-input - [0:0]
:ufw-user-logging-output - [0:0]
:ufw-user-logging-forward - [0:0]
:ufw-user-limit - [0:0]
:ufw-user-limit-accept - [0:0]
-A INPUT -j LIBVIRT_INP
-A INPUT -j ufw-before-logging-input
-A INPUT -j ufw-before-input
-A INPUT -j ufw-after-input
-A INPUT -j ufw-after-logging-input
-A INPUT -j ufw-reject-input
-A INPUT -j ufw-track-input
-A FORWARD -j LIBVIRT_FWX
-A FORWARD -j LIBVIRT_FWI
-A FORWARD -j LIBVIRT_FWO
-A FORWARD -j ufw-before-logging-forward
-A FORWARD -j ufw-before-forward
-A FORWARD -j ufw-after-forward
-A FORWARD -j ufw-after-logging-forward
-A FORWARD -j ufw-reject-forward
-A FORWARD -j ufw-track-forward
-A OUTPUT -j LIBVIRT_OUT
-A OUTPUT -j ufw-before-logging-output
-A OUTPUT -j ufw-before-output
-A OUTPUT -j ufw-after-output
-A OUTPUT -j ufw-after-logging-output
-A OUTPUT -j ufw-reject-output
-A OUTPUT -j ufw-track-output
-A ufw-before-input -i lo -j ACCEPT
-A ufw-before-input -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny
-A ufw-before-input -m conntrack --ctstate INVALID -j DROP
-A ufw-before-input -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A ufw-before-input -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A ufw-before-input -j ufw-not-local
-A ufw-before-input -d 224.0.0.251/32 -p udp -m udp --dport 5353 -j ACCEPT
-A ufw-before-input -d 239.255.255.250/32 -p udp -m udp --dport 1900 -j ACCEPT
-A ufw-before-input -j ufw-user-input
-A ufw-before-output -o lo -j ACCEPT
-A ufw-before-output -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-output -j ufw-user-output
-A ufw-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A ufw-before-forward -j ufw-user-forward
-A ufw-after-input -p udp -m udp --dport 137 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 138 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp -m tcp --dport 139 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp -m tcp --dport 445 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 67 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 68 -j ufw-skip-to-policy-input
-A ufw-after-input -m addrtype --dst-type BROADCAST -j ufw-skip-to-policy-input
-A ufw-after-logging-input -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-after-logging-forward -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-track-output -p tcp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-track-output -p udp -m conntrack --ctstate NEW -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 68 -j ACCEPT
-A LIBVIRT_FWO -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWI -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWX -i virbr0 -o virbr0 -j ACCEPT
-A ufw-logging-deny -m conntrack --ctstate INVALID -m limit --limit 3/min --limit-burst 10 -j RETURN
-A ufw-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-logging-allow -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW ALLOW] "
-A ufw-skip-to-policy-input -j DROP
-A ufw-skip-to-policy-output -j ACCEPT
-A ufw-skip-to-policy-forward -j DROP
-A ufw-not-local -m addrtype --dst-type LOCAL -j RETURN
-A ufw-not-local -m addrtype --dst-type MULTICAST -j RETURN
-A ufw-not-local -m addrtype --dst-type BROADCAST -j RETURN
-A ufw-not-local -m limit --limit 3/min --limit-burst 10 -j ufw-logging-deny
-A ufw-not-local -j DROP
-A ufw-user-limit -m limit --limit 3/min -j LOG --log-prefix "[UFW LIMIT BLOCK] "
-A ufw-user-limit -j REJECT --reject-with icmp-port-unreachable
-A ufw-user-limit-accept -j ACCEPT
COMMIT
# Completed on Mon Jan 25 15:50:52 2021
# Generated by iptables-save v1.8.5 on Mon Jan 25 15:50:52 2021
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:LIBVIRT_PRT - [0:0]
-A POSTROUTING -j LIBVIRT_PRT
-A LIBVIRT_PRT -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A LIBVIRT_PRT -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
COMMIT
# Completed on Mon Jan 25 15:50:52 2021
# Generated by iptables-save v1.8.5 on Mon Jan 25 15:50:52 2021
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:LIBVIRT_PRT - [0:0]
-A POSTROUTING -j LIBVIRT_PRT
-A LIBVIRT_PRT -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Mon Jan 25 15:50:52 2021

So it looks like UFW or libvirt has removed the LXD rules added for lxdbr0.

Suggest restarting LXD using sudo systemctl reload snap.lxd.daemon and then comparing the iptables rules, and then adding equivalent rules to your firewall so they are restored on boot.

Is there a way to make sure that these rules are being added in the first place, as opposed to assuming that they were removed?

I’m asking because purging LXD snap and initializing again doesn’t add them.
Also, If I understood you correctly, this output doesn’t seem right:

root@HOLODECK:~# systemctl reload snap.lxd.daemon
root@HOLODECK:~# iptables-save | grep -i lxd
root@HOLODECK:~#

Can you show output of:

lxc info | grep 'firewall:'

As it may be that LXD is using nftables and you have a mixed iptables/nftables environment (bad).

That’s why I asked earlier for the output of sudo nft list ruleset as well.

Aha!

root@HOLODECK:~# lxc info | grep 'firewall:'
  firewall: nftables
root@HOLODECK:~# nft list ruleset
Command 'nft' not found, but can be installed with:
apt install nftables
root@HOLODECK:~#

It’s a minimal desktop installation of Groovy, I didn’t touch anything firewall related other than:

ufw enable

As far as I understand it’s still the default way to interact with firewall, regardless of underlying technology.

1 Like

If when LXD started there were no iptables rules active, then it would prefer to use nftables if the kernel is recent enough (and the tool is bundled in the snap).

If after that additional iptables rules are added, then you can end up in a mixed environment.

LXD tries to detect various combinations and make the ‘best’ decision at start up, and after that its own rules it adds can influence its driver choice on subsequent start up.

To further complicate matters, nftables provides an iptables shim compatibility layer, except its not fully compatible with iptables or ebtables, so in that scenario we would prefer nftables too.

The logic is here:

Can you install nftables sudo apt install nftables and then try listing the ruleset again.

I recreated the issue.

So on Groovy, installing ufw and using its default configuration will use the iptables shim to insert nftables rules.

So LXD then uses nftables.

Installing nftables package allows you see both the rules added by UFW and by LXD.

However in its default configuration ufw drops all incoming traffic.

Although LXD has added allow rules for DHCP and DNS, these are still blocked because, as per nftables documentation: https://wiki.nftables.org/wiki-nftables/index.php/Configuring_chains

NOTE : If a packet is accepted and there is another chain, bearing the same hook type and with a later priority, then the packet will subsequently traverse this other chain. Hence, an accept verdict - be it by way of a rule or the default chain policy - isn’t necessarily final. However, the same is not true of packets that are subjected to a drop verdict. Instead, drops take immediate effect, with no further rules or chains being evaluated.

So you can get this working by allowing incoming traffic (which rather defeats the purpose of installing ufw):

 ufw default allow incoming

Or you could add rules equivalent to those added in the lxd chains, to ufw to allow inbound DNS and DHCP.

1 Like

@stgraber because of the way nftables allows multiple base chains to operate on a packet (those that have a netfilter hook in them), even if one of them accepts the packet, it may still be dropped later due to another chain’s rules and policies.

Because we create our own lxd chains with netfilter hooks. This means that if an application is using the iptables nftables shim and creates its own base chains with netfilter hooks and a default drop policy then our rules will not be final and packets will be dropped.

This makes it pretty tricky for two applications that create their own hook chains into netfilter to coexist as neither one can selectively accept traffic if the other has dropped all traffic.

One way around this would be for us to create another non-base chain inside a well-known table (like the one used by the iptables shim) and then add jump rules to the well-known chains it uses to give our rules a chance of accepting the packets before their policy kicks in.

1 Like

@stgraber So for example LXD creates this:

table ip lxd {
	chain in.lxdbr0 {
		type filter hook input priority 0; policy accept;
		iifname "lxdbr0" tcp dport 53 accept
		iifname "lxdbr0" udp dport 53 accept
		iifname "lxdbr0" udp dport 67 accept
	}

	chain out.lxdbr0 {
		type filter hook output priority 0; policy accept;
		oifname "lxdbr0" tcp sport 53 accept
		oifname "lxdbr0" udp sport 53 accept
		oifname "lxdbr0" udp sport 67 accept
	}

	chain fwd.lxdbr0 {
		type filter hook forward priority 0; policy accept;
		oifname "lxdbr0" accept
		iifname "lxdbr0" accept
	}

	chain pstrt.lxdbr0 {
		type nat hook postrouting priority 100; policy accept;
		ip saddr 10.63.37.0/24 ip daddr != 10.63.37.0/24 masquerade
	}
}

and UFW (via the iptables shim) creates this:

table ip filter {
	chain INPUT {
		type filter hook input priority 0; policy drop;
	}

	chain FORWARD {
		type filter hook forward priority 0; policy drop;
	}

	chain OUTPUT {
		type filter hook output priority 0; policy accept;
	}
}

We would need to add a rule to the filter table in the INPUT and FOWARD chains to jump into our own chains to stand a chance of the packets being allowed. However nftables doesn’t allow you to jump into a base chain (one that has a netfilter hook in it), and I’m not sure you can jump across tables either.

So to get compatibility with the iptables shim we’d need to add our custom chains to the filter table and then add rules to the INPUT and FOWARD chains to get our rules applied before the DROP policy kicks in.

This would only work with applications that use the filter table, and any other application that creates its own tables and sets up drop policies would cause the same problem again.

Yeah, I’m honestly not sure what’s the right thing to do here…

I’m not super optimistic about us putting workarounds in place to handle the compatibility xtables tooling. By definition this will cause issues as it’s trying to pretend that nft is xtables and so comes with the same issues around rule ordering…

I believe ufw was natively ported to nft recently so it may instead be better to see how we handle the rules generated by that version.

Cooperating properly with other native nft users and pushing distros to ship the native nft support in those tools when available feels like a more future proof way to handle this.

2 Likes

As you can see getting the various firewall implementations to play nicely together is a non-trivial task.

In the meantime these commands seems to suffice to allow traffic from lxdbr0 interface to the LXD host and for traffic from lxdbr0 to be routed to the external network without allowing all external inbound traffic:

sudo ufw allow in on lxdbr0
sudo ufw route allow in on lxdbr0
6 Likes

@tomp Thank you for getting to the bottom of that issue, I would have never solve it myself :slight_smile:

Not quite. In my case, it makes container get IPv4 address. But then, inside the container, I still have networking issues.

root@test:~# apt update
Err:1 http://archive.ubuntu.com/ubuntu focal InRelease                                                                     
  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8001::24). - connect (101: 
Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 
(2001:67c:1360:8001::23). - connect (101: Network is unreachable) Could not connect to 
archive.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to 
archive.ubuntu.com:80 (91.189.88.152), connection timed out
[...snap...]

root@test:~# ip -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128 
eth0@if14        UP             10.52.135.83/24 fe80::216:3eff:feb3:15db/64

root@test:~# ping archive.ubuntu.com
PING archive.ubuntu.com (91.189.88.142) 56(84) bytes of data.
64 bytes from aerodent.canonical.com (91.189.88.142): icmp_seq=1 ttl=50 time=58.3 ms
64 bytes from aerodent.canonical.com (91.189.88.142): icmp_seq=2 ttl=50 time=66.4 ms
^C

root@test:~# curl -m 30 archive.ubuntu.com
curl: (28) Connection timed out after 30000 milliseconds
1 Like

Ah you also need to allow routed traffic to traverse from lxdbr0 to the external network using:

sudo ufw route allow in on lxdbr0

Works like a charm

1 Like

FYI, the ufw deb will follow update-alternatives so if it is setup for iptables-nft, then lxd using nftables is fine. IMHO, this should be true of any firewall software provided via the distro (ie, they should follow update-alternatives) and they should all agree to use xtables or nftables. AIUI, the problem here is that lxd was creating rules on the system when no firewall was in place and when the firewall was enabled, the nft backend was not setup as the default. To get out of this situation, one should be able to use the update-alternatives mechanism to use iptables-nft, then reboot (ufw will then use the nft backend (ie, there is nothing more to be done with ufw). A more complicated series of commands could be done to skip the reboot; I’m not sure if lxd can be made to use iptables-legacy/xtables after the fact).

Note 1: the ufw snap will use a similar algorithm as lxd, so it wouldn’t have this problem.

Note 2: the needed extra ufw route rules/etc are expected.

LXD will prefer using nftables if there are any nftables rules active (including its own) even if there are also xtables legacy rules active.

So to force LXD to go back to xtables legacy (assuming there are already xtables rules present) is to run:

sudo nft flush ruleset
sudo systemctl reload snap.lxd.daemon