LXD Networking problems, help! (docker)

I have several containers running Ubuntu 20.04.

I am able to connect to the containers via SSH. I can also update the the containers using apt.

The problem is that I can no longer make connections to other hosts from within the containers, I think since I upgraded to LXD 5.1.

When I try to ping any external ip address it fails.

How can I diagnose this problem?

lxc info focal
Name: focal
Status: RUNNING
Type: container
Architecture: x86_64
PID: 65544
Created: 2020/11/23 10:16 GMT
Last Used: 2022/05/11 10:42 BST

Resources:
  Processes: 113
  CPU usage:
    CPU usage (in seconds): 30
  Memory usage:
    Memory (current): 511.61MiB
    Memory (peak): 543.86MiB
    Swap (current): 1.11MiB
    Swap (peak): 112.00KiB
  Network usage:
    eth0:
      Type: broadcast
      State: UP
      Host interface: veth747c6e80
      MAC address: 00:16:3e:1d:31:00
      MTU: 1500
      Bytes received: 169.74kB
      Bytes sent: 157.94kB
      Packets received: 1842
      Packets sent: 1733
      IP addresses:
        inet:  10.165.251.237/24 (global)
        inet6: fd42:8121:aac9:ec67:216:3eff:fe1d:3100/64 (global)
        inet6: fe80::216:3eff:fe1d:3100/64 (link)
    lo:
      Type: loopback
      State: UP
      MTU: 65536
      Bytes received: 65.31kB
      Bytes sent: 65.31kB
      Packets received: 823
      Packets sent: 823
      IP addresses:
        inet:  127.0.0.1/8 (local)
        inet6: ::1/128 (local)

I am wondering if this problem could be related to IP6? The host does not have IP6 configured.

Certain kinds of connection are working. For example, I can use nslookup from within a container:

root@focal:~# nslookup google.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	google.com
Address: 216.58.212.238
Name:	google.com
Address: 2a00:1450:4009:80b::200e

But if I try to ping an IP address it fails:

root@focal:~# ping -c 10 216.58.212.238
PING 216.58.212.238 (216.58.212.238) 56(84) bytes of data.

--- 216.58.212.238 ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 9203ms

lxc network list

+---------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
|  NAME   |   TYPE   | MANAGED |      IPV4       |           IPV6            | DESCRIPTION | USED BY |  STATE  |
+---------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
| docker0 | bridge   | NO      |                 |                           |             | 0       |         |
+---------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
| lxdbr0  | bridge   | YES     | 10.165.251.1/24 | fd42:8121:aac9:ec67::1/64 |             | 11      | CREATED |
+---------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
| wlp59s0 | physical | NO      |                 |                           |             | 0       |         |
+---------+----------+---------+-----------------+---------------------------+-------------+---------+---------+

ip a on host

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: wlp59s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9c:b6:d0:ba:f3:33 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.2/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp59s0
       valid_lft 83888sec preferred_lft 83888sec
    inet6 2a00:23a8:8e3:a901:5c3a:c062:1c85:d819/64 scope global temporary dynamic 
       valid_lft 230sec preferred_lft 50sec
    inet6 2a00:23a8:8e3:a901:497e:c0ed:75ac:bbed/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 230sec preferred_lft 50sec
    inet6 fe80::85e5:8a5e:77ae:6e7b/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:ad:b9:c5:80 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
5: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:4d:ab:16 brd ff:ff:ff:ff:ff:ff
    inet 10.165.251.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:8121:aac9:ec67::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe4d:ab16/64 scope link 
       valid_lft forever preferred_lft forever
11: veth5b3afa58@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 76:ee:91:5e:bb:d5 brd ff:ff:ff:ff:ff:ff link-netnsid 2
15: veth747c6e80@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether b2:bd:55:94:80:dd brd ff:ff:ff:ff:ff:ff link-netnsid 0

ip a in container

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
14: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:1d:31:00 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.165.251.237/24 brd 10.165.251.255 scope global dynamic eth0
       valid_lft 2310sec preferred_lft 2310sec
    inet6 fd42:8121:aac9:ec67:216:3eff:fe1d:3100/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 3127sec preferred_lft 3127sec
    inet6 fe80::216:3eff:fe1d:3100/64 scope link 
       valid_lft forever preferred_lft forever

Any help is very gratefully received. I have been using LXD for many years with no issues, but I am not very familiar with Linux networking configuration. :smiling_face:

Sounds like a firewall issue.

Please show sudo iptables-save and sudo nft list ruleset (if available).

Thanks for the reply!

sudo iptables-save

# Generated by iptables-save v1.8.4 on Wed May 11 11:36:11 2022
*nat
:PREROUTING ACCEPT [3958:598378]
:INPUT ACCEPT [2464:496382]
:OUTPUT ACCEPT [11162:937095]
:POSTROUTING ACCEPT [10899:900182]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Wed May 11 11:36:11 2022
# Generated by iptables-save v1.8.4 on Wed May 11 11:36:11 2022
*filter
:INPUT ACCEPT [617076:721790590]
:FORWARD DROP [1276:86774]
:OUTPUT ACCEPT [401992:48551234]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Wed May 11 11:36:11 2022

sudo nft list ruleset not available.

Ah, just saw:

Please see LXD and Docker Firewall Redux - How to deal with FORWARD policy set to drop - #3 by tomp

and we also have a section here:

Please let us know if that helps.

1 Like

Thanks @tomp it was docker causing the issue. Thanks so much for your help :grinning: :pray:

1 Like