Ipv4 broken as of today

21

I used tcpdump -vnes0 -i br0 port 67 or port 68

$ sudo ebtables -L
Bridge table: filter

Bridge chain: INPUT, entries: 0, policy: ACCEPT

Bridge chain: FORWARD, entries: 0, policy: ACCEPT

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
22

Your iptables output shows that something (likely libvirt or docker looking at the rules) have configured a firewall. Interestingly the FORWARD chain has a default DROP rule (DROP 149 packets) and is dropping some packets. I’m not certain that is the issue, however as a first step I would encourage you disable all firewalls (if it is safe to do so) by running iptables -F. Note you may need to restart the machine so that the rules are reinstated afterwards.

23

So DHCPv6 doesn’t use port 67 or port 68 (https://en.wikipedia.org/wiki/DHCPv6#Port_numbers), so you would not see it with that filter anyway.

24

Also, to get a full view of iptables rules can you run iptables-save as you can use that to restore the config later too.

25

Hmm ok I can clear the tables, but upon rebooting it all comes back somehow. grep -r iptables /etc returns nothing useful, so there’s some other part of the system that’s doing this. I don’t want any kind of firewall on this machine at all.

But even if I clear iptables and don’t reboot so that it stays cleared, containers still don’t get an ipv4 address. They just keep firing off DHCPDISCOVER messages with no reply.

26

Can you show the output of iptables-save please.

27

This smells like Docker blocking LXD traffic. We’ve seen it a couple of times before and it gets worse if nft/xtables are mixed.

An option would be to flip the policy on the tables back to ACCEPT and see if the issue goes away. Docker may need to be configured not to completely mess with the system’s firewall…

28

Its quite odd, we’ve tried doing iptables -F, iptables -t mangle -F and iptables -t nat -F to no avail.

29

In the past at least the FORWARD table was the issue, but this may be a different one.

30

I’ve removed all iptables rules (such that iptables-save is empty), its like the bridge isn’t forwarding packets except it is forwarding ARP packets.

If I set the IP statically inside the container and then try to ping the gateway, the gateway response to ARP requests, then we see ICMP requests go out to br0 but not eno1 (the external interface).

31

Did you try setting all of /proc/sys/net/bridge to 0 if not already the case?

32

OK I fixed it.

It was the default DROP policy rule I identified earlier in this thread.

But iptables -F doesn’t reset it apparently.

You have to do iptables -P FORWARD ACCEPT to reset it.

This combined with the following sysctls meant that all bridged traffic goes through iptables FORWARD chain and gets dropped:

net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
33

So to fix this, either:

  1. Ensure your default iptables FORWARD policy is ACCEPT
  2. Disable bridge traffic going through iptables by setting those sysctls to 0.
34

Wow thanks so much!

But I can’t help wonder what got me into this mess to begin with? This is a very generic ubuntu install, and all I did was install a few common server packages (docker, lxd, libvirt). Maybe it was something in the docker version of the deb package?

35

Yes as @stgraber mentioned its likely that something in docker is setting that policy.

36

Docker also sets the policy for the FORWARD chain to DROP . If your Docker host also acts as a router, this will result in that router not forwarding any traffic anymore. If you want your system to continue functioning as a router, you can add explicit ACCEPT rules to the DOCKER-USER chain to allow it:

$ iptables -I DOCKER-USER -i src_if -o dst_if -j ACCEPT

That combined with the default sysctls:

net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

means that bridged traffic also goes through the iptables FORWARD chain.

37

Good to know! Thank you.