Lxd bridge doesn't work with IPv4 and UFW with nftables

Yeah, I’m honestly not sure what’s the right thing to do here…

I’m not super optimistic about us putting workarounds in place to handle the compatibility xtables tooling. By definition this will cause issues as it’s trying to pretend that nft is xtables and so comes with the same issues around rule ordering…

I believe ufw was natively ported to nft recently so it may instead be better to see how we handle the rules generated by that version.

Cooperating properly with other native nft users and pushing distros to ship the native nft support in those tools when available feels like a more future proof way to handle this.

3 Likes

As you can see getting the various firewall implementations to play nicely together is a non-trivial task.

In the meantime these commands seems to suffice to allow traffic from lxdbr0 interface to the LXD host and for traffic from lxdbr0 to be routed to the external network without allowing all external inbound traffic:

sudo ufw allow in on lxdbr0
sudo ufw route allow in on lxdbr0
sudo ufw route allow out on lxdbr0

See our page in the docs about this also:

17 Likes

@tomp Thank you for getting to the bottom of that issue, I would have never solve it myself :slight_smile:

Not quite. In my case, it makes container get IPv4 address. But then, inside the container, I still have networking issues.

root@test:~# apt update
Err:1 http://archive.ubuntu.com/ubuntu focal InRelease                                                                     
  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8001::24). - connect (101: 
Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 
(2001:67c:1360:8001::23). - connect (101: Network is unreachable) Could not connect to 
archive.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to 
archive.ubuntu.com:80 (91.189.88.152), connection timed out
[...snap...]

root@test:~# ip -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128 
eth0@if14        UP             10.52.135.83/24 fe80::216:3eff:feb3:15db/64

root@test:~# ping archive.ubuntu.com
PING archive.ubuntu.com (91.189.88.142) 56(84) bytes of data.
64 bytes from aerodent.canonical.com (91.189.88.142): icmp_seq=1 ttl=50 time=58.3 ms
64 bytes from aerodent.canonical.com (91.189.88.142): icmp_seq=2 ttl=50 time=66.4 ms
^C

root@test:~# curl -m 30 archive.ubuntu.com
curl: (28) Connection timed out after 30000 milliseconds
1 Like

Ah you also need to allow routed traffic to traverse from lxdbr0 to the external network using:

sudo ufw route allow in on lxdbr0
1 Like

Works like a charm

1 Like

FYI, the ufw deb will follow update-alternatives so if it is setup for iptables-nft, then lxd using nftables is fine. IMHO, this should be true of any firewall software provided via the distro (ie, they should follow update-alternatives) and they should all agree to use xtables or nftables. AIUI, the problem here is that lxd was creating rules on the system when no firewall was in place and when the firewall was enabled, the nft backend was not setup as the default. To get out of this situation, one should be able to use the update-alternatives mechanism to use iptables-nft, then reboot (ufw will then use the nft backend (ie, there is nothing more to be done with ufw). A more complicated series of commands could be done to skip the reboot; I’m not sure if lxd can be made to use iptables-legacy/xtables after the fact).

Note 1: the ufw snap will use a similar algorithm as lxd, so it wouldn’t have this problem.

Note 2: the needed extra ufw route rules/etc are expected.

1 Like

LXD will prefer using nftables if there are any nftables rules active (including its own) even if there are also xtables legacy rules active.

So to force LXD to go back to xtables legacy (assuming there are already xtables rules present) is to run:

sudo nft flush ruleset
sudo systemctl reload snap.lxd.daemon
1 Like

Uninstalling UFW and reinstalling it with snap seems to have resolved the issue for me.

sudo ufw disable
sudo apt remove ufw
sudo apt purge ufw
sudo snap install ufw
sudo ufw enable

I can confirm that uninstalling UFW and reinstalling with snap resolves the issue of “No IPV4 addresses assigned to containers”. I just tested on a fresh installation of Ubuntu 21.10. No IPs assigned to new containers until after UFW was reinstalled with snap.

1 Like

This may be because the ufw snap is using its own bundled iptables legacy backend rather than iptables nft backend used via the ufw apt package (because iptables on the host is actually using nftables). I don’t know for sure, but based on what we do with the LXD snap (we bundle both tools), I suspect that may not be a long-term solution.

1 Like

that worked, thx

2 Likes

it worked for me also, LXD running on Fedora 35. Thanks a lot.

1 Like

Sorry about reviving an old thread, but I just want to add some helpful tip here. If you are using Ubuntu 22.04 and nat=true in port forwarding, I’ve discovered that you’ll also need:

ufw route allow out on lxdbr0

Otherwise, the latest ufw blocks traffic that tries to get out on lxdbr0 (which is only true if nat=true is used in forwarding a port). Also, the rule above is kinda too permissive - you may want to specify ports that are relevant. :slight_smile:

3 Likes

When I upgraded from 20.04 to 22.04 Jammy, lxd stopped working. It didn’t hand out IPv4 addresses. This solution worked for me. Thank you.

1 Like

yes this helped me when i upgraded the host but when i upgrade a container from Debian 10 to 11 this solution stopped work for me. I tried disabling reseting reinstalling with snap version the ufw firewall also tried to disable lxd firewall and still doesn’t get ipv4 from lxdbr0 to eth0 into container (btw i also tried to use macvlan type instead of nic or whatever still nothing) also i uninstall docker to prevent conflict or whatever still not getting this f ipv4 address

After finishing

sudo lxd init

Could the devs please give link to this https://linuxcontainers.org/lxd/docs/master/howto/network_bridge_firewalld/#ufw-add-rules-for-the-bridge

sudo ufw allow in on lxdbr0
sudo ufw route allow in on lxdbr0
sudo ufw route allow out on lxdbr0

for all ubuntu snap installations. I was scratching my mind for ages until I found this.

Please can you open an issue https://github.com/lxc/lxd/issues so your idea can be discussed further.

For posterity and to perhaps help with the docs, I found that these finer-grained rules seem to work well when ufw is enabled, iptables-nft/ip6tables-nft are in use and when ipv4.nat: "true" (seen with lxc network show lxdbr0):

# for a guest to get an ip from the lxd host
$ sudo ufw allow in on lxdbr0 to any port 67 proto udp comment 'lxd: dhcp'

# for a guest to resolve hostnames from the lxd host
$ sudo ufw allow in on lxdbr0 to any port 53 comment 'lxd: dns/resolved'

# for a guest to have access to anything outbound (10.210.254.0/24
# found with: lxc network list | grep lxdbr0)
$ sudo ufw route allow in on lxdbr0 from 10.210.254.0/24 comment 'lxd: outbound'

The first two rules limit access from the guests to the host to only dhcp and dns while the 3rd rule allows access from the guests to the world (the last rule mimics the default behavior of ufw for the host, which is to allow all outbound (from ipv4 hosts; adjust if using ipv6)). Adjust as necessary for your environment.

Note, the above doesn’t have have a fine-grained equivalent of sudo ufw route allow out on lxdbr0. If you are wanting to allow traffic from outside the host to your guests, then additional rules are in order (eg, sudo ufw route allow out on lxdbr0 to 10.210.254.123 port 443 proto tcp comment "lxd: inbound to https server").

1 Like

Hello dear @tomp and community. I am new in this forum and I am desperately looking for a solution for my lxd implemantation.

2 years ago a installed lxd containers on ubuntu 18.04 host.

Launching images, creating bridges on the host and configuring ip addresses on the containers was very easy.
I had some services like dns, web server and other working as containers and could access to outside internet without problem. They also could talk each other. I was using new multiple bridges br0, br1 and br3 instead of lxdbr0. No additional firewall configuration on the host was necessary to set.
Unfortunately hade hardware problem and needed to install a new one.

The problem is that with ubuntu 22.04 server the new container are not communicating to outside the the host.

I did followed the basic proceedure as previouly:

  1. install bridge-utils
    #apt-get install bridge-utils
  2. Config bridges on host netplan file
  3. sudo apt update
    #sudo apt upgrade
  4. Initialize lxd
    #sudo lxd init
  5. Creating containers
    #sudo lxc launch ubuntu:22.04 webserver
  6. Config network interfaces inside the containers
    /etc/netplan/ → [ etho,eth1,eth2 ]
  7. Add interfaces to containers for Host bridges
    #sudo lxc config device add webserver eth0 nic name=eth0 nictype=bridged parent=br0

#------------------------------
Host
#-----------------------------
OS: ubuntu 22
IP 10.0.0.2/24
gateway: 10.0.0.1
nameservers 10.0.0.1, 8.8.8.8
Interfaces: ens1, ens2, ens3 physical
br0 (attached to ens1), br1 (to ens2), br2(to ens3)
can host access internet : yes
can host ping containers inside the host (same network): yes
can host ping machines outside the host (same network): yes

#-----------------------------
Guest - Container (CT)
#----------------------------
OS: ubuntu 22, name: webserver - created as lxc launch ubuntu:22.04 webserver
IP 10.0.0.10/24
gateway 10.0.0.1
nameservers 10.0.0.1,8.8.8.8
Interfaces: ens1, ens2, ens3 physical
br0 (attached to ens1), br1 (to ens2), br2(to ens3)
can CT access internet : no
can CT ping other CTs inside the host (same network): yes
can CT ping other machines outside the host (same network): no

#OS: ubuntu 20, name: local-dns
IP 10.0.0.11/24
The rest is same as previous

Can anyone help me with this issue or point to a previouly?

  1. What am I not doing correctly?
  2. What additional configuration is required?
  3. If this issue has been answered can you please please post the link?
  4. Let me know if you require additional information
    Many thanks