Containers can't access other hosts in the LAN or Internet also running docker

LXD
,
1

Hello all,

I am having this annoying problem. The host is a laptop running Ubuntu 20.04 LTS, configured with a bridge interface: br0 and using the physical eno1 interface as slave. I have connectivity just fine to the LAN and Internet from the LXD host.

Bridge interface status:

kumo2:~$ ip a s dev br0
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 5c:26:0a:40:47:c9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.12/24 brd 192.168.0.255 scope global br0
       valid_lft forever preferred_lft forever

I can ping the internet:

kumo2:~$ ping -c 5 cisco.com
PING cisco.com (72.163.4.185) 56(84) bytes of data.
64 bytes from redirect-ns.cisco.com (72.163.4.185): icmp_seq=1 ttl=231 time=163 ms
64 bytes from redirect-ns.cisco.com (72.163.4.185): icmp_seq=2 ttl=231 time=157 ms
64 bytes from redirect-ns.cisco.com (72.163.4.185): icmp_seq=3 ttl=231 time=155 ms
64 bytes from redirect-ns.cisco.com (72.163.4.185): icmp_seq=4 ttl=231 time=166 ms
64 bytes from redirect-ns.cisco.com (72.163.4.185): icmp_seq=5 ttl=231 time=161 ms

--- cisco.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 155.049/160.169/165.505/3.742 ms

Default profile configuration:

kumo2:~$ sudo lxc profile show default
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    pool: DATA1
    type: disk
name: default
used_by:
- /1.0/instances/openstack

Container status and configured with static IP address:

kumo2:~$ sudo lxc list
+-----------+---------+---------------------+--------------------------------------------+-----------+-----------+
|   NAME    |  STATE  |        IPV4         |                    IPV6                    |   TYPE    | SNAPSHOTS |
+-----------+---------+---------------------+--------------------------------------------+-----------+-----------+
| openstack | RUNNING | 192.168.0.41 (eth0) | fdca:4894:ac20::5c8 (eth0)                 | CONTAINER | 0         |
|           |         |                     | fdca:4894:ac20:0:216:3eff:feb2:6c33 (eth0) |           |           |
+-----------+---------+---------------------+--------------------------------------------+-----------+-----------+

Network list:

kumo2:~$ sudo lxc network list
+---------+----------+---------+------+------+-------------+---------+
|  NAME   |   TYPE   | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY |
+---------+----------+---------+------+------+-------------+---------+
| br0     | bridge   | NO      |      |      |             | 2       |
+---------+----------+---------+------+------+-------------+---------+
| docker0 | bridge   | NO      |      |      |             | 0       |
+---------+----------+---------+------+------+-------------+---------+
| eno1    | physical | NO      |      |      |             | 0       |
+---------+----------+---------+------+------+-------------+---------+
| wlp2s0  | physical | NO      |      |      |             | 0       |
+---------+----------+---------+------+------+-------------+---------+

Container current configuration:

kumo2:~$ sudo lxc config show openstack
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 20.04 LTS amd64 (release) (20220207)
  image.label: release
  image.os: ubuntu
  image.release: focal
  image.serial: "20220207"
  image.type: squashfs
  image.version: "20.04"
  volatile.base_image: 017d8572a250a681aec28aa59da770d95db2b79ba156dd590d5f250d0df974ea
  volatile.eth0.host_name: veth02527063
  volatile.eth0.hwaddr: 00:16:3e:b2:6c:33
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 859e36ad-7b2c-41f7-a99b-aca6b7998f98
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

Any obvious detail here that I should check or try?

Below is the ICMP test from the LXD host pinging the container, tcpdump capture added as well.

kumo2:~$ ping -c 3 192.168.0.41
PING 192.168.0.41 (192.168.0.41) 56(84) bytes of data.
64 bytes from 192.168.0.41: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 192.168.0.41: icmp_seq=2 ttl=64 time=0.090 ms
64 bytes from 192.168.0.41: icmp_seq=3 ttl=64 time=0.089 ms

--- 192.168.0.41 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2053ms
rtt min/avg/max/mdev = 0.089/0.099/0.118/0.013 ms

openstack:~# tcpdump -i eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
07:43:40.554231 IP 192.168.0.12 > openstack: ICMP echo request, id 5, seq 1, length 64
07:43:40.554279 IP openstack > 192.168.0.12: ICMP echo reply, id 5, seq 1, length 64
07:43:41.583370 IP 192.168.0.12 > openstack: ICMP echo request, id 5, seq 2, length 64
07:43:41.583410 IP openstack > 192.168.0.12: ICMP echo reply, id 5, seq 2, length 64
07:43:42.607309 IP 192.168.0.12 > openstack: ICMP echo request, id 5, seq 3, length 64
07:43:42.607348 IP openstack > 192.168.0.12: ICMP echo reply, id 5, seq 3, length 64

Here is the ICMP test from a host on the same LAN pinging the LXD hos, tcpdump capture added as well.

host1:~$ ping -c 3 192.168.0.12
PING 192.168.0.12 (192.168.0.12) 56(84) bytes of data.
64 bytes from 192.168.0.12: icmp_seq=1 ttl=64 time=1.06 ms
64 bytes from 192.168.0.12: icmp_seq=2 ttl=64 time=0.435 ms
64 bytes from 192.168.0.12: icmp_seq=3 ttl=64 time=0.408 ms

--- 192.168.0.12 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2020ms
rtt min/avg/max/mdev = 0.408/0.633/1.057/0.299 ms

kumo2:~$ sudo tcpdump -i br0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:46:25.463313 IP 192.168.0.21 > kumo2.ddns.net: ICMP echo request, id 4277, seq 1, length 64
16:46:25.463634 IP kumo2.ddns.net > 192.168.0.21: ICMP echo reply, id 4277, seq 1, length 64
16:46:26.463238 IP 192.168.0.21 > kumo2.ddns.net: ICMP echo request, id 4277, seq 2, length 64
16:46:26.463343 IP kumo2.ddns.net > 192.168.0.21: ICMP echo reply, id 4277, seq 2, length 64
16:46:27.483308 IP 192.168.0.21 > kumo2.ddns.net: ICMP echo request, id 4277, seq 3, length 64
16:46:27.483384 IP kumo2.ddns.net > 192.168.0.21: ICMP echo reply, id 4277, seq 3, length 64
16:46:47.970641 IP _gateway > kumo2.ddns.net: ICMP _gateway udp port 5351 unreachable, length 48

Here is the ICMP test from a host on the same LAN pinging the LXD container, tcpdump capture added as well.

host1:~$ ping -c 3 192.168.0.41
PING 192.168.0.41 (192.168.0.41) 56(84) bytes of data.

--- 192.168.0.41 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2052ms

The container didn’t capture any packages…

Any suggestions will be appreciated, I am thinking in re-installing and running again the lxd init command.

Sincerely,

2

Are you running the LXD host inside a VM that applies MAC filtering to its external NIC, or is there anything on the network that would enforce a single MAC address for your external LAN connection port?

3

@tomp none of those.

It is a simple home LAN connected to a home dumb router, no mac address or access lists restrictions of any kind, no ubuntu UFW not even active.

kumo2:~$ sudo ufw status
Status: inactive

kumo2:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp – anywhere anywhere multiport dports mdns
ACCEPT tcp – anywhere anywhere multiport dports 4000

Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all – anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all – anywhere anywhere
ACCEPT all – anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all – anywhere anywhere
ACCEPT all – anywhere anywhere
ACCEPT all – anywhere anywhere

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain DOCKER (1 references)
target prot opt source destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
RETURN all – anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all – anywhere anywhere
RETURN all – anywhere anywhere

Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all – anywhere anywhere

This suppose to work in my opinion, is there a was for me to run $ sudo lxd init again, the only thing different on the initial set up is, that when running $ sudo lxd init I selected not to create and manage a bridge. I created the bridge br0 after the $ sudo lxd init was completed, as you can recall I thought I could bridge the wireless adapter and that was not possible, so I created the bridge with the Ethernet adapter and edited the default profile and changed the options in bold below:

config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: DATA1
type: disk
name: default
used_by:

  • /1.0/instances/openstack

That was all. Thank you so much for reviewing this with me, it is really strange in my opinion.

Hmmm…

4

Ah you’re also running docker, this is well known on these forums for breaking LXD’s bridging by modifying the firewall rules (outside of ufw).

See Lxd and Docker Firewall Redux - How to deal with FORWARD policy set to drop - #3 by tomp

1 Like
5

Hello @tomp let me check on that and test the IPTABLES rules you mentioned and what I can find in the reference post.

I will update.

Sincerely,

6

Hello @tomp awesome! that was it!

Here is the test after applying this: iptables -I DOCKER-USER -j ACCEPT

From a host in the LAN:

host1~$ ping -c 5 192.168.0.41
PING 192.168.0.41 (192.168.0.41) 56(84) bytes of data.
64 bytes from 192.168.0.41: icmp_seq=1 ttl=64 time=0.829 ms
64 bytes from 192.168.0.41: icmp_seq=2 ttl=64 time=0.423 ms
64 bytes from 192.168.0.41: icmp_seq=3 ttl=64 time=0.339 ms
64 bytes from 192.168.0.41: icmp_seq=4 ttl=64 time=0.532 ms
64 bytes from 192.168.0.41: icmp_seq=5 ttl=64 time=0.498 ms

--- 192.168.0.41 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4085ms
rtt min/avg/max/mdev = 0.339/0.524/0.829/0.166 ms

TCPDUMP capture on the lxd host bridge interface br0:

kumo2:~$ sudo tcpdump -i br0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:54:23.339035 IP 192.168.0.21 > 192.168.0.41: ICMP echo request, id 4482, seq 1, length 64
22:54:23.339217 IP 192.168.0.41 > 192.168.0.21: ICMP echo reply, id 4482, seq 1, length 64
22:54:24.355751 IP 192.168.0.21 > 192.168.0.41: ICMP echo request, id 4482, seq 2, length 64
22:54:24.355848 IP 192.168.0.41 > 192.168.0.21: ICMP echo reply, id 4482, seq 2, length 64
22:54:25.375654 IP 192.168.0.21 > 192.168.0.41: ICMP echo request, id 4482, seq 3, length 64
22:54:25.375741 IP 192.168.0.41 > 192.168.0.21: ICMP echo reply, id 4482, seq 3, length 64
22:54:26.399874 IP 192.168.0.21 > 192.168.0.41: ICMP echo request, id 4482, seq 4, length 64
22:54:26.399957 IP 192.168.0.41 > 192.168.0.21: ICMP echo reply, id 4482, seq 4, length 64
22:54:27.423852 IP 192.168.0.21 > 192.168.0.41: ICMP echo request, id 4482, seq 5, length 64
22:54:27.423951 IP 192.168.0.41 > 192.168.0.21: ICMP echo reply, id 4482, seq 5, length 64

TCPDUMP capture on the lxc container interface eth0:

openstack:~# tcpdump -i eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
13:54:23.339108 IP 192.168.0.21 > openstack: ICMP echo request, id 4482, seq 1, length 64
13:54:23.339205 IP openstack > 192.168.0.21: ICMP echo reply, id 4482, seq 1, length 64
13:54:24.355780 IP 192.168.0.21 > openstack: ICMP echo request, id 4482, seq 2, length 64
13:54:24.355845 IP openstack > 192.168.0.21: ICMP echo reply, id 4482, seq 2, length 64
13:54:25.375679 IP 192.168.0.21 > openstack: ICMP echo request, id 4482, seq 3, length 64
13:54:25.375737 IP openstack > 192.168.0.21: ICMP echo reply, id 4482, seq 3, length 64
13:54:26.399898 IP 192.168.0.21 > openstack: ICMP echo request, id 4482, seq 4, length 64
13:54:26.399945 IP openstack > 192.168.0.21: ICMP echo reply, id 4482, seq 4, length 64
13:54:27.423882 IP 192.168.0.21 > openstack: ICMP echo request, id 4482, seq 5, length 64
13:54:27.423947 IP openstack > 192.168.0.21: ICMP echo reply, id 4482, seq 5, length 64

It works now… I spent almost a full day trying to figure out why it was not working :face_with_head_bandage: good to know, thank you so much! this should be definitely in one of the must FAQ or banner in the forum, warning if you are running docker see this! :wink:

Sincerely,

2 Likes