Containers do not have outgoing internet access

debashis · April 22, 2021, 8:52am

Hi,

I was happily using LXD containers for a couple of years now, recently I have noticed problem accessing internet from within my containers, restarting lxd and restarting the host machine previously resolved the problem. But this time nothing is helping, I have even tried uninstalling lxd and re-installing it.

My system:
Host: Ubuntu 20.04
LXD version 4.13

I am using all default configuration values during the installation, nothing is special.

$ lxc ls

+-----------+---------+----------------------+-----------------------------------------------+-----------+-----------+
|   NAME    |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+-----------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| ubuntuone | RUNNING | 10.235.48.173 (eth0) | fd42:a188:52d4:442d:216:3eff:fe8f:9757 (eth0) | CONTAINER | 0         |
+-----------+---------+----------------------+-----------------------------------------------+-----------+-----------+

$ lxc exec ubuntuone -- ping -c1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

debashis · April 22, 2021, 8:54am

Network on the host machine:

root@debashis-ThinkPad:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether e8:6a:64:89:01:31 brd ff:ff:ff:ff:ff:ff
3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 20:79:18:5f:4d:bc brd ff:ff:ff:ff:ff:ff
    inet 192.168.87.20/24 brd 192.168.87.255 scope global dynamic noprefixroute wlp2s0
       valid_lft 46677sec preferred_lft 46677sec
    inet6 fe80::57d:6815:66b4:c320/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: enx3c18a0ba710c: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 3c:18:a0:ba:71:0c brd ff:ff:ff:ff:ff:ff
6: br-2c0eaa4592c2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:66:2e:07:40 brd ff:ff:ff:ff:ff:ff
    inet 172.26.0.1/16 brd 172.26.255.255 scope global br-2c0eaa4592c2
       valid_lft forever preferred_lft forever
7: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:50:85:c8:87 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
11: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:c9:45:47 brd ff:ff:ff:ff:ff:ff
    inet 10.235.48.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:a188:52d4:442d::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fec9:4547/64 scope link 
       valid_lft forever preferred_lft forever
13: vethf473a9d8@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 32:ab:5f:19:85:b1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
root@debashis-ThinkPad:~# ip r
default via 192.168.87.1 dev wlp2s0 proto dhcp metric 600 
10.235.48.0/24 dev lxdbr0 proto kernel scope link src 10.235.48.1 
169.254.0.0/16 dev wlp2s0 scope link metric 1000 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.26.0.0/16 dev br-2c0eaa4592c2 proto kernel scope link src 172.26.0.1 linkdown 
192.168.87.0/24 dev wlp2s0 proto kernel scope link src 192.168.87.20 metric 600

debashis · April 22, 2021, 8:56am

iptables on the host machine:

root@debashis-ThinkPad:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  10.1.0.0/16          anywhere             /* generated for MicroK8s pods */
ACCEPT     all  --  anywhere             10.1.0.0/16          /* generated for MicroK8s pods */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (2 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

debashis · April 22, 2021, 2:14pm

I applied the solution of this other thread Lxd uses iptables-legacy even if iptables-nft is default and that solved my problem

which is basically to flush all the iptable rules. This is all I did -

# for ipt in iptables iptables-legacy ip6tables ip6tables-legacy; do $ipt --flush; $ipt --flush -t nat; $ipt --delete-chain; $ipt --delete-chain -t nat; $ipt -P FORWARD ACCEPT; $ipt -P INPUT ACCEPT; $ipt -P OUTPUT ACCEPT; done
# systemctl reload snap.lxd.daemon

And that was it

perlun · September 13, 2022, 1:59pm

For the record, I’ve seen this exact problem as well with recent lxd using the Snap package installed by following the instructions at Linux Containers - LXD - Getting started. I’ve been generally annoyed with having to use snapd for this so I’ve recently switched over to the (not yet Debian-included) upcoming lxd package for Debian from https://apt.calenhad.com/

Interestingly enough, I’ve had the exact same problem with lxd installed from that package though. After email conversation with Mathias Gibbens (thanks!), the root cause seems to be the iptables package on my system, causing lxd to be confused. And iptables is in turn installed because it’s a dependency by the docker.io package, i.e. Docker…

So the problem will probably go away if you uninstall docker.io, but until then the workaround described above can perhaps be of help for other people who also use Docker + LXD on the same machine.

tomp · September 15, 2022, 3:02pm

See How to configure your firewall - LXD documentation

hassan_Bodal · April 19, 2023, 9:50am

Thanks ! that worked

perlun · January 5, 2024, 1:51pm

I know that the LXD project has now moved to Canonical’s forum, but in case someone finds this thread and struggles with LXD+Docker networking: here is the updated URL now: How to configure your firewall - Canonical LXD documentation

perlun · January 23, 2024, 8:14am

And for those of you wondering about Docker+Incus, this page seems to have the up-to-date information: How to configure your firewall - Incus documentation