Containers can't access other hosts in the LAN or Internet also running docker

Hello all,

I am having this annoying problem. The host is a laptop running Ubuntu 20.04 LTS, configured with a bridge interface: br0 and using the physical eno1 interface as slave. I have connectivity just fine to the LAN and Internet from the LXD host.

Bridge interface status:

kumo2:~$ ip a s dev br0
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 5c:26:0a:40:47:c9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.12/24 brd 192.168.0.255 scope global br0
       valid_lft forever preferred_lft forever

I can ping the internet:

kumo2:~$ ping -c 5 cisco.com
PING cisco.com (72.163.4.185) 56(84) bytes of data.
64 bytes from redirect-ns.cisco.com (72.163.4.185): icmp_seq=1 ttl=231 time=163 ms
64 bytes from redirect-ns.cisco.com (72.163.4.185): icmp_seq=2 ttl=231 time=157 ms
64 bytes from redirect-ns.cisco.com (72.163.4.185): icmp_seq=3 ttl=231 time=155 ms
64 bytes from redirect-ns.cisco.com (72.163.4.185): icmp_seq=4 ttl=231 time=166 ms
64 bytes from redirect-ns.cisco.com (72.163.4.185): icmp_seq=5 ttl=231 time=161 ms

--- cisco.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 155.049/160.169/165.505/3.742 ms

Default profile configuration:

kumo2:~$ sudo lxc profile show default
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    pool: DATA1
    type: disk
name: default
used_by:
- /1.0/instances/openstack

Container status and configured with static IP address:

kumo2:~$ sudo lxc list
+-----------+---------+---------------------+--------------------------------------------+-----------+-----------+
|   NAME    |  STATE  |        IPV4         |                    IPV6                    |   TYPE    | SNAPSHOTS |
+-----------+---------+---------------------+--------------------------------------------+-----------+-----------+
| openstack | RUNNING | 192.168.0.41 (eth0) | fdca:4894:ac20::5c8 (eth0)                 | CONTAINER | 0         |
|           |         |                     | fdca:4894:ac20:0:216:3eff:feb2:6c33 (eth0) |           |           |
+-----------+---------+---------------------+--------------------------------------------+-----------+-----------+

Network list:

kumo2:~$ sudo lxc network list
+---------+----------+---------+------+------+-------------+---------+
|  NAME   |   TYPE   | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY |
+---------+----------+---------+------+------+-------------+---------+
| br0     | bridge   | NO      |      |      |             | 2       |
+---------+----------+---------+------+------+-------------+---------+
| docker0 | bridge   | NO      |      |      |             | 0       |
+---------+----------+---------+------+------+-------------+---------+
| eno1    | physical | NO      |      |      |             | 0       |
+---------+----------+---------+------+------+-------------+---------+
| wlp2s0  | physical | NO      |      |      |             | 0       |
+---------+----------+---------+------+------+-------------+---------+

Container current configuration:

kumo2:~$ sudo lxc config show openstack
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 20.04 LTS amd64 (release) (20220207)
  image.label: release
  image.os: ubuntu
  image.release: focal
  image.serial: "20220207"
  image.type: squashfs
  image.version: "20.04"
  volatile.base_image: 017d8572a250a681aec28aa59da770d95db2b79ba156dd590d5f250d0df974ea
  volatile.eth0.host_name: veth02527063
  volatile.eth0.hwaddr: 00:16:3e:b2:6c:33
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 859e36ad-7b2c-41f7-a99b-aca6b7998f98
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

Any obvious detail here that I should check or try?

Below is the ICMP test from the LXD host pinging the container, tcpdump capture added as well.

kumo2:~$ ping -c 3 192.168.0.41
PING 192.168.0.41 (192.168.0.41) 56(84) bytes of data.
64 bytes from 192.168.0.41: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 192.168.0.41: icmp_seq=2 ttl=64 time=0.090 ms
64 bytes from 192.168.0.41: icmp_seq=3 ttl=64 time=0.089 ms

--- 192.168.0.41 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2053ms
rtt min/avg/max/mdev = 0.089/0.099/0.118/0.013 ms
openstack:~# tcpdump -i eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
07:43:40.554231 IP 192.168.0.12 > openstack: ICMP echo request, id 5, seq 1, length 64
07:43:40.554279 IP openstack > 192.168.0.12: ICMP echo reply, id 5, seq 1, length 64
07:43:41.583370 IP 192.168.0.12 > openstack: ICMP echo request, id 5, seq 2, length 64
07:43:41.583410 IP openstack > 192.168.0.12: ICMP echo reply, id 5, seq 2, length 64
07:43:42.607309 IP 192.168.0.12 > openstack: ICMP echo request, id 5, seq 3, length 64
07:43:42.607348 IP openstack > 192.168.0.12: ICMP echo reply, id 5, seq 3, length 64

Here is the ICMP test from a host on the same LAN pinging the LXD hos, tcpdump capture added as well.

host1:~$ ping -c 3 192.168.0.12
PING 192.168.0.12 (192.168.0.12) 56(84) bytes of data.
64 bytes from 192.168.0.12: icmp_seq=1 ttl=64 time=1.06 ms
64 bytes from 192.168.0.12: icmp_seq=2 ttl=64 time=0.435 ms
64 bytes from 192.168.0.12: icmp_seq=3 ttl=64 time=0.408 ms

--- 192.168.0.12 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2020ms
rtt min/avg/max/mdev = 0.408/0.633/1.057/0.299 ms
kumo2:~$ sudo tcpdump -i br0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:46:25.463313 IP 192.168.0.21 > kumo2.ddns.net: ICMP echo request, id 4277, seq 1, length 64
16:46:25.463634 IP kumo2.ddns.net > 192.168.0.21: ICMP echo reply, id 4277, seq 1, length 64
16:46:26.463238 IP 192.168.0.21 > kumo2.ddns.net: ICMP echo request, id 4277, seq 2, length 64
16:46:26.463343 IP kumo2.ddns.net > 192.168.0.21: ICMP echo reply, id 4277, seq 2, length 64
16:46:27.483308 IP 192.168.0.21 > kumo2.ddns.net: ICMP echo request, id 4277, seq 3, length 64
16:46:27.483384 IP kumo2.ddns.net > 192.168.0.21: ICMP echo reply, id 4277, seq 3, length 64
16:46:47.970641 IP _gateway > kumo2.ddns.net: ICMP _gateway udp port 5351 unreachable, length 48

Here is the ICMP test from a host on the same LAN pinging the LXD container, tcpdump capture added as well.

host1:~$ ping -c 3 192.168.0.41
PING 192.168.0.41 (192.168.0.41) 56(84) bytes of data.

--- 192.168.0.41 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2052ms

The container didn’t capture any packages…

Any suggestions will be appreciated, I am thinking in re-installing and running again the lxd init command.

Sincerely,

Are you running the LXD host inside a VM that applies MAC filtering to its external NIC, or is there anything on the network that would enforce a single MAC address for your external LAN connection port?

@tomp none of those.

It is a simple home LAN connected to a home dumb router, no mac address or access lists restrictions of any kind, no ubuntu UFW not even active.

kumo2:~$ sudo ufw status
Status: inactive

kumo2:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp – anywhere anywhere multiport dports mdns
ACCEPT tcp – anywhere anywhere multiport dports 4000

Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all – anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all – anywhere anywhere
ACCEPT all – anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all – anywhere anywhere
ACCEPT all – anywhere anywhere
ACCEPT all – anywhere anywhere

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain DOCKER (1 references)
target prot opt source destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
RETURN all – anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all – anywhere anywhere
RETURN all – anywhere anywhere

Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all – anywhere anywhere

This suppose to work in my opinion, is there a was for me to run $ sudo lxd init again, the only thing different on the initial set up is, that when running $ sudo lxd init I selected not to create and manage a bridge. I created the bridge br0 after the $ sudo lxd init was completed, as you can recall I thought I could bridge the wireless adapter and that was not possible, so I created the bridge with the Ethernet adapter and edited the default profile and changed the options in bold below:

config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: DATA1
type: disk
name: default
used_by:

  • /1.0/instances/openstack

That was all. Thank you so much for reviewing this with me, it is really strange in my opinion.

Hmmm…

Ah you’re also running docker, this is well known on these forums for breaking LXD’s bridging by modifying the firewall rules (outside of ufw).

See Lxd and Docker Firewall Redux - How to deal with FORWARD policy set to drop - #3 by tomp

1 Like

Hello @tomp let me check on that and test the IPTABLES rules you mentioned and what I can find in the reference post.

I will update.

Sincerely,

Hello @tomp awesome! that was it!

Here is the test after applying this: iptables -I DOCKER-USER -j ACCEPT

From a host in the LAN:

host1~$ ping -c 5 192.168.0.41
PING 192.168.0.41 (192.168.0.41) 56(84) bytes of data.
64 bytes from 192.168.0.41: icmp_seq=1 ttl=64 time=0.829 ms
64 bytes from 192.168.0.41: icmp_seq=2 ttl=64 time=0.423 ms
64 bytes from 192.168.0.41: icmp_seq=3 ttl=64 time=0.339 ms
64 bytes from 192.168.0.41: icmp_seq=4 ttl=64 time=0.532 ms
64 bytes from 192.168.0.41: icmp_seq=5 ttl=64 time=0.498 ms

--- 192.168.0.41 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4085ms
rtt min/avg/max/mdev = 0.339/0.524/0.829/0.166 ms

TCPDUMP capture on the lxd host bridge interface br0:

kumo2:~$ sudo tcpdump -i br0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:54:23.339035 IP 192.168.0.21 > 192.168.0.41: ICMP echo request, id 4482, seq 1, length 64
22:54:23.339217 IP 192.168.0.41 > 192.168.0.21: ICMP echo reply, id 4482, seq 1, length 64
22:54:24.355751 IP 192.168.0.21 > 192.168.0.41: ICMP echo request, id 4482, seq 2, length 64
22:54:24.355848 IP 192.168.0.41 > 192.168.0.21: ICMP echo reply, id 4482, seq 2, length 64
22:54:25.375654 IP 192.168.0.21 > 192.168.0.41: ICMP echo request, id 4482, seq 3, length 64
22:54:25.375741 IP 192.168.0.41 > 192.168.0.21: ICMP echo reply, id 4482, seq 3, length 64
22:54:26.399874 IP 192.168.0.21 > 192.168.0.41: ICMP echo request, id 4482, seq 4, length 64
22:54:26.399957 IP 192.168.0.41 > 192.168.0.21: ICMP echo reply, id 4482, seq 4, length 64
22:54:27.423852 IP 192.168.0.21 > 192.168.0.41: ICMP echo request, id 4482, seq 5, length 64
22:54:27.423951 IP 192.168.0.41 > 192.168.0.21: ICMP echo reply, id 4482, seq 5, length 64

TCPDUMP capture on the lxc container interface eth0:

openstack:~# tcpdump -i eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
13:54:23.339108 IP 192.168.0.21 > openstack: ICMP echo request, id 4482, seq 1, length 64
13:54:23.339205 IP openstack > 192.168.0.21: ICMP echo reply, id 4482, seq 1, length 64
13:54:24.355780 IP 192.168.0.21 > openstack: ICMP echo request, id 4482, seq 2, length 64
13:54:24.355845 IP openstack > 192.168.0.21: ICMP echo reply, id 4482, seq 2, length 64
13:54:25.375679 IP 192.168.0.21 > openstack: ICMP echo request, id 4482, seq 3, length 64
13:54:25.375737 IP openstack > 192.168.0.21: ICMP echo reply, id 4482, seq 3, length 64
13:54:26.399898 IP 192.168.0.21 > openstack: ICMP echo request, id 4482, seq 4, length 64
13:54:26.399945 IP openstack > 192.168.0.21: ICMP echo reply, id 4482, seq 4, length 64
13:54:27.423882 IP 192.168.0.21 > openstack: ICMP echo request, id 4482, seq 5, length 64
13:54:27.423947 IP openstack > 192.168.0.21: ICMP echo reply, id 4482, seq 5, length 64

It works now… I spent almost a full day trying to figure out why it was not working :face_with_head_bandage: good to know, thank you so much! this should be definitely in one of the must FAQ or banner in the forum, warning if you are running docker see this! :wink:

Sincerely,

2 Likes