Lxdbr0 firewall problem with Ubuntu 22.04 host running Docker and LXD

LXD
1

I have a brand-new Ubuntu 22.04 VPS host and have installed LXD (via snap) and Docker (via Install Docker Engine on Ubuntu | Docker Documentation) and now LXD containers are not able to ping the outside world.

The exact same setup on Ubuntu 20.04 VPS host it all works perfectly out of the box and there are iptables rules defined with comments like /* generated for LXD network lxdbr0 */, that I suppose allow it to play nicely with the rules added by Docker.

However on Ubuntu 22.04 host it does not work out of the box and I need to add the following workaround to get it to work:

# iptables -A FORWARD -i lxdbr0 -j ACCEPT
# iptables -A FORWARD -o lxdbr0 -j ACCEPT

I guess I can add that workaround permanently, but it seems like I shouldn’t need to? My understanding is that because network lxdbr0 config ipv4.firewall is not set it will default to true and should be creating these iptables rules. Why is it for Ubuntu 20 and isn’t it for Ubuntu 22?

2

This is most likely caused by docker see

3

It’s definitely caused by Docker. I tried the command suggested in that link: iptables -I DOCKER-USER -i lxdbr0 -o eth0 -j ACCEPT and it did not work.

The two commands I listed above did work. As did uninstalling Docker (apt purge docker-ce docker-ce-cli containerd.io docker-compose-plugin) and rebooting.

Interestingly when I reinstall Docker, the lxd container networking is fine until after I reboot the host, after which it stops working.

4

This is most likely a race condition between LXD and Docker starting up, which explains the inconsistency when it works if the start order is changed.

What does lxc info | grep firewall: show?

5 
root@jammy:~# lxc info | grep firewall
- network_firewall_filtering
- firewall_driver
  firewall: nftables
6

Try adding an empty firewall rule to iptables, such as iptables -A INPUT ensuring that is present before LXD is started. Then restart your system and see if LXD then uses iptables rather than nftables.

That might be enough to get docker and LXD to play nicely.

7

Didn’t work.

This seems to be a duplicate of this issue How do I configure persistent firewall settings in Ubuntu 22.04 LTS - confused about ufw, nfw and iptables(-nfw/-legacy) - Ask Ubuntu

8

Well lets check first. Did you confirm that lxc info | grep firewall: shows xtables?

9

We also have a section on ufw

10 
root@jammy:~# ufw status
Status: inactive
11

And the firewall driver?

12

It worked but had to use iptables-legacy to add the dummy rule:

root@jammy:~# lxc info | grep firewall:
  firewall: nftables
root@jammy:~# snap stop lxd
2022-10-07T21:32:44+10:00 INFO Waiting for "snap.lxd.daemon.service" to stop.
Stopped.
root@jammy:~# iptables -A INPUT
root@jammy:~# snap start lxd
Started.
root@jammy:~# lxc info | grep firewall:
  firewall: nftables
root@jammy:~# snap stop lxd
2022-10-07T21:33:45+10:00 INFO Waiting for "snap.lxd.daemon.service" to stop.
Stopped.
root@jammy:~# iptables -F
root@jammy:~# iptables-legacy -A INPUT
root@jammy:~# snap start lxd
Started.
root@jammy:~# lxc info | grep firewall:
  firewall: xtables
root@jammy:~# reboot

But not persistent after reboot:

root@jammy:~# lxc info | grep firewall:
  firewall: nftables
13

Also noted that after installing Docker, but before the first reboot, all of the docker iptables rules are listed under iptables-legacy -L. At this stage LXD and Docker play well together. But then after reboot the docker iptables rules move to iptables -L.

14

You could try removing nftables.

15

The comment above about docker using iptables-legacy before/after a reboot was wrong.

It seems the solution is to just use Docker from a Snap. If you already have docker from apt, purge it, install it from snap and reboot.

I redeployed a fresh VM, which has lxd pre-installed. I then install docker via apt and notice it’s using nftables:

root@jammy:~# apt install docker.io
...
root@jammy:~# nft list ruleset | grep -i docker | wc -l
17
root@jammy:~# iptables-legacy -L | grep -i docker | wc -l
0

and after a reboot LXD networking is not working while both share nftables:

root@jammy:~# lxc info | grep firewall:
  firewall: nftables
root@jammy:~# lxc exec foo -- ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2036ms

But if I replace docker from apt with docker from snap and then reboot:

root@jammy:~# apt purge docker.io
...
root@jammy:~# apt autoremove
...
root@jammy:~# snap install docker
...
root@jammy:~# reboot

following reboot LXD external networking is working because the snap docker seems to keep to using iptables-legacy while lxd uses nftables:

root@jammy:~# lxc exec foo -- ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=12.1 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=12.1 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.109/12.127/12.145/0.018 ms
root@jammy:~# iptables-legacy -L | grep -i docker | wc -l
8
root@jammy:~# nft list ruleset | grep -i docker | wc -l
0