Docker container doesn't have internet when running inside of LXD container

thanethomson · November 19, 2021, 1:28am

In a nutshell my problem seems to be related to networking and container nesting. Here’s the hierarchy of components and where the problem is:

(1) Ubuntu 20.04.3 LTS (has internet )
- (2) LXD
  - (3) Debian bullseye container (has internet )
    - (4) Docker CE
      - (5) ubuntu:latest container (has no internet )

From (1): the host

> sudo iptables -vL

Chain INPUT (policy ACCEPT 20719 packets, 43M bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     icmp --  lxdbr0 any     anywhere             anywhere             icmp parameter-problem /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     icmp --  lxdbr0 any     anywhere             anywhere             icmp time-exceeded /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     icmp --  lxdbr0 any     anywhere             anywhere             icmp destination-unreachable /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     tcp  --  lxdbr0 any     anywhere             anywhere             tcp dpt:domain /* generated for LXD network lxdbr0 */
  110  6640 ACCEPT     udp  --  lxdbr0 any     anywhere             anywhere             udp dpt:domain /* generated for LXD network lxdbr0 */
   18  5864 ACCEPT     udp  --  lxdbr0 any     anywhere             anywhere             udp dpt:bootps /* generated for LXD network lxdbr0 */
21743   43M LIBVIRT_INP  all  --  any    any     anywhere             anywhere            
  692 87721 NFQUEUE    udp  --  any    any     anywhere             anywhere             udp spt:domain NFQUEUE num 0 bypass

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
11326  227M ACCEPT     all  --  any    lxdbr0  anywhere             anywhere             /* generated for LXD network lxdbr0 */
10471  577K ACCEPT     all  --  lxdbr0 any     anywhere             anywhere             /* generated for LXD network lxdbr0 */
    0     0 LIBVIRT_FWX  all  --  any    any     anywhere             anywhere            
    0     0 LIBVIRT_FWI  all  --  any    any     anywhere             anywhere            
    0     0 LIBVIRT_FWO  all  --  any    any     anywhere             anywhere            

Chain OUTPUT (policy ACCEPT 24515 packets, 4360K bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     icmp --  any    lxdbr0  anywhere             anywhere             icmp parameter-problem /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     icmp --  any    lxdbr0  anywhere             anywhere             icmp time-exceeded /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     icmp --  any    lxdbr0  anywhere             anywhere             icmp destination-unreachable /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     tcp  --  any    lxdbr0  anywhere             anywhere             tcp spt:domain /* generated for LXD network lxdbr0 */
  110 11392 ACCEPT     udp  --  any    lxdbr0  anywhere             anywhere             udp spt:domain /* generated for LXD network lxdbr0 */
   15  4986 ACCEPT     udp  --  any    lxdbr0  anywhere             anywhere             udp spt:bootps /* generated for LXD network lxdbr0 */
24980 4417K LIBVIRT_OUT  all  --  any    any     anywhere             anywhere            

Chain LIBVIRT_FWI (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  any    virbr0  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED
    0     0 REJECT     all  --  any    virbr0  anywhere             anywhere             reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  virbr0 any     192.168.122.0/24     anywhere            
    0     0 REJECT     all  --  virbr0 any     anywhere             anywhere             reject-with icmp-port-unreachable

Chain LIBVIRT_FWX (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  virbr0 virbr0  anywhere             anywhere            

Chain LIBVIRT_INP (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  virbr0 any     anywhere             anywhere             udp dpt:domain
    0     0 ACCEPT     tcp  --  virbr0 any     anywhere             anywhere             tcp dpt:domain
    0     0 ACCEPT     udp  --  virbr0 any     anywhere             anywhere             udp dpt:bootps
    0     0 ACCEPT     tcp  --  virbr0 any     anywhere             anywhere             tcp dpt:67

Chain LIBVIRT_OUT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  any    virbr0  anywhere             anywhere             udp dpt:domain
    0     0 ACCEPT     tcp  --  any    virbr0  anywhere             anywhere             tcp dpt:domain
    0     0 ACCEPT     udp  --  any    virbr0  anywhere             anywhere             udp dpt:bootpc
    0     0 ACCEPT     tcp  --  any    virbr0  anywhere             anywhere             tcp dpt:68

To launch (3) - Debian LXD container:

lxc launch -c security.nesting=true images:debian/bullseye dockertest
lxc exec dockertest -- /bin/bash

Inside (3) - Debian LXD container, as `root`

# ping test
> ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=3.74 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=54 time=4.44 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=54 time=4.18 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=54 time=2.33 ms
^C
--- 1.1.1.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3006ms
rtt min/avg/max/mdev = 2.327/3.672/4.443/0.815 ms

# Install iptables, Docker-CE dependencies
> apt update && apt install -y ca-certificates curl gnupg lsb-release iptables
> curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
> echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
# Install Docker-CE
> apt update && apt install -y docker-ce docker-ce-cli containerd.io

# Double-check that internet still works (since I've seen in other places that
# Docker can mess up iptables forwarding rules, but it's fine for me here)
> ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=3.75 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=54 time=4.33 ms
^C
--- 1.1.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 3.748/4.039/4.331/0.291 ms

Some networking config from within the Debian LXD container:

> iptables -vL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-USER  all  --  any    any     anywhere             anywhere            
    0     0 DOCKER-ISOLATION-STAGE-1  all  --  any    any     anywhere             anywhere            
    0     0 ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  any    docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  docker0 !docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  anywhere             anywhere            
    0     0 RETURN     all  --  any    any     anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  any    docker0  anywhere             anywhere            
    0     0 RETURN     all  --  any    any     anywhere             anywhere            

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  any    any     anywhere             anywhere

> ifconfig

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:1c:0d:bb:9b  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.114.76.158  netmask 255.255.255.0  broadcast 10.114.76.255
        inet6 fd42:1544:5012:6eba:216:3eff:fe25:dd5f  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::216:3eff:fe25:dd5f  prefixlen 64  scopeid 0x20<link>
        ether 00:16:3e:25:dd:5f  txqueuelen 1000  (Ethernet)
        RX packets 7764  bytes 143383313 (136.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7096  bytes 496110 (484.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

> cat /etc/resolv.conf
domain lxd
search lxd
nameserver 10.114.76.1

Back on (1) - the host system

lxc list dockertest
+------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
|    NAME    |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| dockertest | RUNNING | 172.17.0.1 (docker0) | fd42:1544:5012:6eba:216:3eff:fe25:dd5f (eth0) | CONTAINER | 0         |
|            |         | 10.114.76.158 (eth0) |                                               |           |           |
+------------+---------+----------------------+-----------------------------------------------+-----------+-----------+

lxc network show lxdbr0
config:
  ipv4.address: 10.114.76.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:1544:5012:6eba::1/64
  ipv6.nat: "true"
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/dockertest
- /1.0/profiles/default
managed: true
status: Created
locations:
- none

Now back to (3) - Debian LXD container as `root`

# This works as expected, giving me an interactive prompt as root in an
# Ubuntu container.
docker run -it --rm ubuntu /bin/bash

Inside (5) - Docker `ubuntu:latest` container as `root`

> apt update

Err:1 http://archive.ubuntu.com/ubuntu focal InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease           
  Temporary failure resolving 'archive.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease         
  Temporary failure resolving 'archive.ubuntu.com'
Err:4 http://security.ubuntu.com/ubuntu focal-security InRelease
  Temporary failure resolving 'security.ubuntu.com'
Reading package lists... Done         
Building dependency tree       
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.

> cat /etc/resolv.conf
domain lxd
search lxd
nameserver 10.114.76.1

Any ideas as to how I can get the internet working from within the ubuntu:latest Docker-CE container? I’ve tried with several other Docker containers and the result’s always the same - no access to the underlying network whatsoever. In another Alpine-based container with ping preinstalled, I can’t even get a basic ping operation to work:

> ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
ping: can't create raw socket: Permission denied

> ifconfig
ifconfig: socket(AF_INET,2,0): Permission denied

placid_chat · November 19, 2021, 6:28am

My lxd host [ docker host [ host A ] ] works fine. Originally apparmor was complaining, but "security.nesting true " helped with that.

thanethomson · November 19, 2021, 4:31pm

It looks as though this may have something to do with the Debian LXD container. When I repeat the above, but for an Ubuntu LXD container, the ubuntu Docker container has internet.

capriciousduck · October 9, 2023, 10:12am

I too see the same issue. Have you found a solution?

Docker container doesn't have internet when running inside of LXD container

From (1): the host

Inside (3) - Debian LXD container, as root

Back on (1) - the host system

Now back to (3) - Debian LXD container as root

Inside (5) - Docker ubuntu:latest container as root

Inside (3) - Debian LXD container, as `root`

Now back to (3) - Debian LXD container as `root`

Inside (5) - Docker `ubuntu:latest` container as `root`