While there are a lot of discussions regards LXD problems associated with Docker setting the FORWARD policy to drop and thus preventing the container from being to access the internet, there isn’t - or I haven’t found - a working example of how to fix the problems.
One solution is to add the following rule: iptables -I DOCKER-USER -i src_if -o dst_if -j ACCEPT
But I have no idea on how to determine the src_if or the dst_if.
I’ve seen other posts with other solutions which either don’t work or don’t make sense if one is not well versed in iptables.
Can someone post a canonical answer including an real world example using real world interfaces?
In your case the -i <src_if> src_if would be your lxdbr0 interface and -o dst_if would be your external interface.
However you could relax this further and just use:
iptables -I DOCKER-USER -j ACCEPT
To allow all traffic.
However the reason there isn’t a canonical answer for this is that it all depends on your particular networking setup and your security stance (allowing all traffic maybe not what you want for example).
Then reboot so both are using iptables rather than nftables.
Take a look at iptables rules:
iptables-save
# Generated by iptables-save v1.8.4 on Mon Jan 18 12:45:12 2021
*mangle
:PREROUTING ACCEPT [119:46747]
:INPUT ACCEPT [107:45739]
:FORWARD ACCEPT [12:1008]
:OUTPUT ACCEPT [132:15000]
:POSTROUTING ACCEPT [144:16008]
-A POSTROUTING -o lxdbr0 -p udp -m udp --dport 68 -m comment --comment "generated for LXD network lxdbr0" -j CHECKSUM --checksum-fill
COMMIT
# Completed on Mon Jan 18 12:45:12 2021
# Generated by iptables-save v1.8.4 on Mon Jan 18 12:45:12 2021
*filter
:INPUT ACCEPT [105:45123]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [130:14342]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A INPUT -i lxdbr0 -p tcp -m tcp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 67 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -o lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -o lxdbr0 -p tcp -m tcp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 67 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Mon Jan 18 12:45:12 2021
# Generated by iptables-save v1.8.4 on Mon Jan 18 12:45:12 2021
*nat
:PREROUTING ACCEPT [3:470]
:INPUT ACCEPT [1:302]
:OUTPUT ACCEPT [27:2228]
:POSTROUTING ACCEPT [26:2188]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 10.249.25.0/24 ! -d 10.249.25.0/24 -m comment --comment "generated for LXD network lxdbr0" -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Mon Jan 18 12:45:12 2021
They key lines here are:
This is where docker has modified the FORWARD policy to drop all unmatched traffic:
:FORWARD DROP [0:0]
But here we can see that docker has added a catch-all jump into the DOCKER-USER chain, but before that LXD has added allow rules for traffic for lxdbr0.
-A FORWARD -o lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -j DOCKER-USER
In my case this means that networking is running OK. So it may just be a case of ensuring LXD starts before Docker. However it would be interesting to see the output of iptables-save in your case when it isn’t working. I suspect there may be a race between docker and LXD rule ordering that can affect it.
Note that the current solution in the docs does not work on Debian (bookworm). What worked is making it bi-directional. I documented it in the github issue.
Sure - would you prefer an additional debian-specific paragraph in the docs, leaving the existing instruction as is, or adding the reverse direction to the existing instruction?
When I had to run docker on a router some time ago I ended up putting the docker daemon into it’s own network namespace so it can’t mess anything up and you don’t have to disable iptables management for the docker daemon which might cause other issues.
With that you simply get a veth interface on the host that you can configure using your favorite firewall(e.g. plain nftables)
for anyone hitting this… docker looks for net.ipv4.ip_forward=1 and if that is set it doesn’t touch the default FORWARD policy… there is a detailled discussion about the issue here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=865975
LXD does enable that on bridge networks, but if Docker has started before LXD and that setting isn’t enabled at the system level then that could explain why Docker is not detecting it set at the time it starts and is modifying the FORWARD policy.