Networking not working in container with docker

Daniel · January 17, 2021, 8:12pm

For some reason, outbound connectivity isn’t working in my containers:

root@build05:~# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
^C
--- 1.1.1.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2040ms

The host can ping the container, the container can ping the host, but the container can’t reach the outside Internet.

Network config on host:

root@chi03:~# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UP group default qlen 1000
    link/ether 00:16:3c:c0:ef:fa brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    altname ens3
    inet xxx.xxx.xxx.xxx/24 brd xxx.xxx.xxx.xxx.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 xxxxxxxxx::2/64 scope global
       valid_lft forever preferred_lft forever
    inet6 xxxxxxxxxx:effa/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq state UP group default qlen 1000
    link/ether 00:18:3e:7a:98:1e brd ff:ff:ff:ff:ff:ff
    altname enp0s4
    altname ens4
    inet 10.xxx.xxx.xxx/8 brd 10.255.255.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 xxxxxxxxxxxxxxxx:981e/64 scope link
       valid_lft forever preferred_lft forever
4: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 8920 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none
    inet 10.123.1.41/32 scope global wg0
       valid_lft forever preferred_lft forever
5: br-6b057de625dc: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:c7:9a:1a:1d brd ff:ff:ff:ff:ff:ff
    inet 172.21.0.1/16 brd 172.21.255.255 scope global br-6b057de625dc
       valid_lft forever preferred_lft forever
6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:69:3c:07:8e brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
7: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:42:8d:aa brd ff:ff:ff:ff:ff:ff
    inet 10.121.186.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:de7e:c573:b57b::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe42:8daa/64 scope link
       valid_lft forever preferred_lft forever
9: vethd366efd3@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 8a:8e:b8:4b:5e:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0
11: vethfc34dffa@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 9e:c7:8a:85:ad:ea brd ff:ff:ff:ff:ff:ff link-netnsid 1

lxc config:

root@chi03:~# lxc network list
+-----------------+----------+---------+-----------------+---------------------------+-------------+---------+
|      NAME       |   TYPE   | MANAGED |      IPV4       |           IPV6            | DESCRIPTION | USED BY |
+-----------------+----------+---------+-----------------+---------------------------+-------------+---------+
| br-6b057de625dc | bridge   | NO      |                 |                           |             | 0       |
+-----------------+----------+---------+-----------------+---------------------------+-------------+---------+
| docker0         | bridge   | NO      |                 |                           |             | 0       |
+-----------------+----------+---------+-----------------+---------------------------+-------------+---------+
| eth0            | physical | NO      |                 |                           |             | 0       |
+-----------------+----------+---------+-----------------+---------------------------+-------------+---------+
| eth1            | physical | NO      |                 |                           |             | 0       |
+-----------------+----------+---------+-----------------+---------------------------+-------------+---------+
| lxdbr0          | bridge   | YES     | 10.121.186.1/24 | fd42:de7e:c573:b57b::1/64 |             | 3       |
+-----------------+----------+---------+-----------------+---------------------------+-------------+---------+

root@chi03:~# lxc network show lxdbr0
config:
  ipv4.address: 10.121.186.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:de7e:c573:b57b::1/64
  ipv6.nat: "true"
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/build05
- /1.0/instances/tv
- /1.0/profiles/default
managed: true
status: Created
locations:
- none

Network in container:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
8: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:d0:d7:0a brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.121.186.3/24 brd 10.121.186.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fd42:de7e:c573:b57b:216:3eff:fed0:d70a/64 scope global mngtmpaddr dynamic
       valid_lft 3181sec preferred_lft 3181sec
    inet6 fe80::216:3eff:fed0:d70a/64 scope link
       valid_lft forever preferred_lft forever

Any ideas?

stgraber · January 17, 2021, 8:34pm

We’ve seen this kind of stuff on systems that also have Docker installed.
Docker has a tendency to put very wide filtering rules in place which may interfere with other applications doing bridge networking on the system.

firewalld on some distros can have a similar efefct.

Daniel · January 17, 2021, 8:48pm

I guess installing Docker is what broke it. Any suggestions on how to use Docker and LXC at the same time, with networking working in both?

I’m running this on a KVM virtual server so macvlan is not an option.

stgraber · January 17, 2021, 9:00pm

I believe @tomp posted a link a few weeks ago on this forum about how to configure Docker networking to be less problematic.

In general the main issue is Docker modifying the default policy, so maybe try:

iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT

This should reset the policies to the OS default and may be enough to get things to behave again.

Daniel · January 17, 2021, 9:02pm

Thanks! While you were replying, I found this GitHub issue:

github.com/docker/for-linux

Docker blocking network of existing LXC containers

opened 11:03AM - 15 Sep 17 UTC

thesourcerer8

I am running a server with LXC containers on it. I found a software that was av…ailable as Docker containers, so I installed Docker on the server and tried that software. * [x] This is reporting unexpected problematic behaviour * [x] This is a feature request * [x] I searched existing issues before opening this one ### Expected behavior Both LXC containers should work normally, and new Docker containers should work. If it isn't possible for Docker to start a container without interrupting existing containers due to any conceptual issue, I would expect Docker to detect that such a problem exists and to provide an overridable error message or at least a warning, that it might interfere with existing containers. ### Actual behavior The Docker containers work, but the LXC containers are suddenly disconnected from the network. The LXC containers are still running, and I can attach to them directly, but they aren't visible on the network anymore, they cannot be pinged, and their services aren't available anymore. Docker has disconnected the running LXC containers from the network. ### Steps to reproduce the behavior Install Debian Linux Install LXC Create a LXC container with it's own IP address Run a server on the LXC container Install Docker Download and run a Docker container Try to access the server in the LXC container over the network **Output of `docker version`:** ``` Client: Version: 17.07.0-ce API version: 1.31 Go version: go1.8.3 Git commit: 8784753 Built: Tue Aug 29 17:42:54 2017 OS/Arch: linux/amd64 Server: Version: 17.07.0-ce API version: 1.31 (minimum version 1.12) Go version: go1.8.3 Git commit: 8784753 Built: Tue Aug 29 17:41:44 2017 OS/Arch: linux/amd64 Experimental: false ``` Containers: 1 Running: 0 Paused: 0 Stopped: 1 Images: 4 Server Version: 17.07.0-ce Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 42 Dirperm1 Supported: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 3addd840653146c90a254301d6c3a663c7fd6429 runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4 init version: 949e6fa Security Options: seccomp Profile: default Kernel Version: 4.9.0-3-amd64 Operating System: Debian GNU/Linux 9 (stretch) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 31.41GiB Name: servername ID: ZQY3:PKBB:X7ZY:WY54:VKG4:IILY:42LG:KURS:4MXC:JR67:A6IX:T3KT Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Http Proxy: http://proxy:8080/ Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false WARNING: No swap limit support ``` ``` **Additional environment details (AWS, VirtualBox, physical, etc.)** Linux version 4.9.0-3-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18) ) #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26)

Flushing the FORWARD chain and changing the default policy to ACCEPT was sufficient to get it working again:

iptables -F FORWARD
iptables -P FORWARD ACCEPT

I guess I might have to add a systemd drop-in to run these commands after Docker starts so that it persists after rebooting the system / restarting Docker.

tomp · January 18, 2021, 11:28am

The docker manual has some info and suggestions to fix this: