Lxd and Docker Firewall Redux - How to deal with FORWARD policy set to drop

All:

While there are a lot of discussions regards LXD problems associated with Docker setting the FORWARD policy to drop and thus preventing the container from being to access the internet, there isn’t - or I haven’t found - a working example of how to fix the problems.

One solution is to add the following rule: iptables -I DOCKER-USER -i src_if -o dst_if -j ACCEPT

But I have no idea on how to determine the src_if or the dst_if.

I’ve seen other posts with other solutions which either don’t work or don’t make sense if one is not well versed in iptables.

Can someone post a canonical answer including an real world example using real world interfaces?

Thanks in advance,

-steve

1 Like

I agree it would be useful to have a firewall guide for manual setup.

The network documentation is rather generic (set it to trusted zone: https://github.com/lxc/lxd/blob/master/doc/networks.md#allow-dhcp-dns-with-firewalld ) and includes only firewalld.

The docker manual has some information on preventing their default behaviour of preventing the host operating as a router (which LXD needs).

In your case the -i <src_if> src_if would be your lxdbr0 interface and -o dst_if would be your external interface.

However you could relax this further and just use:

 iptables -I DOCKER-USER  -j ACCEPT

To allow all traffic.

However the reason there isn’t a canonical answer for this is that it all depends on your particular networking setup and your security stance (allowing all traffic maybe not what you want for example).

I’ve just tried installing LXD and Docker side by side in a fresh Ubuntu Focal VM:

Install LXD and test connectivity from a container:

apt install snapd -y; snap install lxd
lxd init --auto
lxc launch images:ubuntu/focal c1
lxc exec c1 -- ping 8.8.8.8

Install docker via instructions at https://docs.docker.com/engine/install/ubuntu/

Then reboot so both are using iptables rather than nftables.

Take a look at iptables rules:

iptables-save
# Generated by iptables-save v1.8.4 on Mon Jan 18 12:45:12 2021
*mangle
:PREROUTING ACCEPT [119:46747]
:INPUT ACCEPT [107:45739]
:FORWARD ACCEPT [12:1008]
:OUTPUT ACCEPT [132:15000]
:POSTROUTING ACCEPT [144:16008]
-A POSTROUTING -o lxdbr0 -p udp -m udp --dport 68 -m comment --comment "generated for LXD network lxdbr0" -j CHECKSUM --checksum-fill
COMMIT
# Completed on Mon Jan 18 12:45:12 2021
# Generated by iptables-save v1.8.4 on Mon Jan 18 12:45:12 2021
*filter
:INPUT ACCEPT [105:45123]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [130:14342]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A INPUT -i lxdbr0 -p tcp -m tcp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 67 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -o lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -o lxdbr0 -p tcp -m tcp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 67 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Mon Jan 18 12:45:12 2021
# Generated by iptables-save v1.8.4 on Mon Jan 18 12:45:12 2021
*nat
:PREROUTING ACCEPT [3:470]
:INPUT ACCEPT [1:302]
:OUTPUT ACCEPT [27:2228]
:POSTROUTING ACCEPT [26:2188]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 10.249.25.0/24 ! -d 10.249.25.0/24 -m comment --comment "generated for LXD network lxdbr0" -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Mon Jan 18 12:45:12 2021

They key lines here are:

This is where docker has modified the FORWARD policy to drop all unmatched traffic:

:FORWARD DROP [0:0]

But here we can see that docker has added a catch-all jump into the DOCKER-USER chain, but before that LXD has added allow rules for traffic for lxdbr0.

-A FORWARD -o lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -j DOCKER-USER

In my case this means that networking is running OK. So it may just be a case of ensuring LXD starts before Docker. However it would be interesting to see the output of iptables-save in your case when it isn’t working. I suspect there may be a race between docker and LXD rule ordering that can affect it.

Hey Tomp,

I just had my internet access from lxc containers stop working today. I also have docker installed along on the host as well as the lxc container

For comparison sake here’s my iptables-save

# Generated by iptables-save v1.8.4 on Mon Jan 18 22:44:14 2021
*raw
:PREROUTING ACCEPT [201740:677885538]
:OUTPUT ACCEPT [185859:40757442]
COMMIT
# Completed on Mon Jan 18 22:44:14 2021
# Generated by iptables-save v1.8.4 on Mon Jan 18 22:44:14 2021
*mangle
:PREROUTING ACCEPT [201740:677885538]
:INPUT ACCEPT [187316:675198813]
:FORWARD ACCEPT [14424:2686725]
:OUTPUT ACCEPT [185859:40757442]
:POSTROUTING ACCEPT [195032:43368572]
COMMIT
# Completed on Mon Jan 18 22:44:14 2021
# Generated by iptables-save v1.8.4 on Mon Jan 18 22:44:14 2021
*nat
:PREROUTING ACCEPT [9589:1101723]
:INPUT ACCEPT [435:108059]
:OUTPUT ACCEPT [12875:886050]
:POSTROUTING ACCEPT [13417:909427]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.5.0.0/16 ! -o br-59f076e4cabf -j MASQUERADE
-A POSTROUTING -s 10.5.0.2/32 -d 10.5.0.2/32 -p tcp -m tcp --dport 5053 -j MASQUERADE
-A POSTROUTING -s 10.5.0.2/32 -d 10.5.0.2/32 -p udp -m udp --dport 5053 -j MASQUERADE
-A POSTROUTING -s 10.5.0.4/32 -d 10.5.0.4/32 -p udp -m udp --dport 51820 -j MASQUERADE
-A POSTROUTING -s 10.5.0.3/32 -d 10.5.0.3/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 10.5.0.3/32 -d 10.5.0.3/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 10.5.0.3/32 -d 10.5.0.3/32 -p tcp -m tcp --dport 53 -j MASQUERADE
-A POSTROUTING -s 10.5.0.3/32 -d 10.5.0.3/32 -p udp -m udp --dport 53 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-59f076e4cabf -j RETURN
-A DOCKER ! -i br-59f076e4cabf -p tcp -m tcp --dport 5053 -j DNAT --to-destination 10.5.0.2:5053
-A DOCKER ! -i br-59f076e4cabf -p udp -m udp --dport 5053 -j DNAT --to-destination 10.5.0.2:5053
-A DOCKER ! -i br-59f076e4cabf -p udp -m udp --dport 51820 -j DNAT --to-destination 10.5.0.4:51820
-A DOCKER ! -i br-59f076e4cabf -p tcp -m tcp --dport 4430 -j DNAT --to-destination 10.5.0.3:443
-A DOCKER ! -i br-59f076e4cabf -p tcp -m tcp --dport 8000 -j DNAT --to-destination 10.5.0.3:80
-A DOCKER ! -i br-59f076e4cabf -p tcp -m tcp --dport 5054 -j DNAT --to-destination 10.5.0.3:53
-A DOCKER ! -i br-59f076e4cabf -p udp -m udp --dport 5054 -j DNAT --to-destination 10.5.0.3:53
COMMIT
# Completed on Mon Jan 18 22:44:14 2021
# Generated by iptables-save v1.8.4 on Mon Jan 18 22:44:14 2021
*filter
:INPUT ACCEPT [187287:675196231]
:FORWARD DROP [7588:383127]
:OUTPUT ACCEPT [185829:40755044]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-59f076e4cabf -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-59f076e4cabf -j DOCKER
-A FORWARD -i br-59f076e4cabf ! -o br-59f076e4cabf -j ACCEPT
-A FORWARD -i br-59f076e4cabf -o br-59f076e4cabf -j ACCEPT
-A DOCKER -d 10.5.0.2/32 ! -i br-59f076e4cabf -o br-59f076e4cabf -p tcp -m tcp --dport 5053 -j ACCEPT
-A DOCKER -d 10.5.0.2/32 ! -i br-59f076e4cabf -o br-59f076e4cabf -p udp -m udp --dport 5053 -j ACCEPT
-A DOCKER -d 10.5.0.4/32 ! -i br-59f076e4cabf -o br-59f076e4cabf -p udp -m udp --dport 51820 -j ACCEPT
-A DOCKER -d 10.5.0.3/32 ! -i br-59f076e4cabf -o br-59f076e4cabf -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 10.5.0.3/32 ! -i br-59f076e4cabf -o br-59f076e4cabf -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 10.5.0.3/32 ! -i br-59f076e4cabf -o br-59f076e4cabf -p tcp -m tcp --dport 53 -j ACCEPT
-A DOCKER -d 10.5.0.3/32 ! -i br-59f076e4cabf -o br-59f076e4cabf -p udp -m udp --dport 53 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-59f076e4cabf ! -o br-59f076e4cabf -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-59f076e4cabf -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Mon Jan 18 22:44:14 2021

I’m not very familiar with IP tables, so not much if I can comment on what’s going here.

It looks like something (docker perhaps) has refreshed the firewall rules and removed the ones LXD added.

Assuming your LXD bridge is lxdbr0 then you’re missing these lines:

-A FORWARD -o lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT

before the line:

-A FORWARD -j DOCKER-USER
1 Like

Great information.

Thanks much!

-steve

I really hate iptables so I use nftables wherever possible. However I found this gist.

Not happy with Docker modifying your precious firewall rules?

Prerequisites

Install Docker CE and nftables:

$ sudo apt-get install nftables
$ sudo systemctl --now enable nftables

Installing

Manually (create/modify daemon.json before starting docker.service):

$ sudo systemctl start docker
$ sudo systemctl stop docker containerd
$ sudo iptables-save > iptables-docker.conf
$ sudo iptables-restore-translate -f iptable-docker.conf > docker.nft
$ sudo nft flush ruleset
$ sudo nft -f docker.nft
$ sudo nft -s list ruleset > /etc/nftables-docker.conf

tl;dr

$ curl -fsSLO https://gist.github.com/goll/bdd6b43c2023f82d15729e9b0067de60/raw/nftables-docker.sh
$ sudo bash -x nftables-docker.sh

For a persistent config just overwrite /etc/nftables.conf with /etc/nftables-docker.conf

If you prefer manual start/stop you can create an alias for example:

alias dock-on='sudo nft -f /etc/nftables-docker.conf && sudo systemctl start docker'
alias dock-off='sudo systemctl stop docker containerd && sudo nft -f /etc/nftables.conf && sudo ip l d docker0'

Then they provide a way to prevent docker from changing firewall settings.

#!/bin/bash
systemctl stop docker containerd
tee /etc/docker/daemon.json << EOF
{
  "iptables": false
}
EOF
curl -fsSLO https://gist.github.com/goll/bdd6b43c2023f82d15729e9b0067de60/raw/nftables-docker.conf -o /etc/nftables-docker.conf && nft -f /etc/nftables-docker.conf
systemctl start docker

So I did this and combined it with the above rule that @tomp wrote giving a ruleset like this:

table ip nat {
	chain PREROUTING {
		type nat hook prerouting priority dstnat; policy accept;
		fib daddr type local counter jump DOCKER
	}

	chain INPUT {
		type nat hook input priority 100; policy accept;
	}

	chain OUTPUT {
		type nat hook output priority -100; policy accept;
		ip daddr != 127.0.0.0/8 fib daddr type local counter jump DOCKER
	}

	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
		oifname != "docker0" ip saddr 172.17.0.0/16 counter masquerade
	}

	chain DOCKER {
		iifname "docker0" counter return
	}
}
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority filter; policy drop;
		oifname "lxdbr0" counter accept comment "generated for LXD network lxdbr0"
		iifname "lxdbr0" counter accept comment "generated for LXD network lxdbr0"
		counter jump DOCKER-USER
		counter jump DOCKER-ISOLATION-STAGE-1
		oifname "docker0" ct state established,related counter accept
		oifname "docker0" counter jump DOCKER
		iifname "docker0" oifname != "docker0" counter accept
		iifname "docker0" oifname "docker0" counter accept
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
	}

	chain DOCKER {
	}

	chain DOCKER-ISOLATION-STAGE-1 {
		iifname "docker0" oifname != "docker0" counter jump DOCKER-ISOLATION-STAGE-2
		counter return
	}

	chain DOCKER-ISOLATION-STAGE-2 {
		oifname "docker0" counter drop
		counter return
	}

	chain DOCKER-USER {
		counter return
	}
}

Note that the rules @tomp mentions:

are these in nftables syntax:

oifname "lxdbr0" counter accept comment "generated for LXD network lxdbr0"
iifname "lxdbr0" counter accept comment "generated for LXD network lxdbr0"

These rules must be above the JUMP rules in the FORWARD chain.