Can't ping anything on the internet from inside containers anymore (since 3.9 I believe)

vrms · January 21, 2019, 6:28am

I am running LXD from the snap (which recently was updated to LXC 3.9)
I access containers only locally on my host machine (debian 9) via the container IP’s (range 10.19.225.xxx)
since the update can not ping anything on the internet (likewise not update anything) with a unknown host linuxcontainers.org error, which worked before, so I kind of suspect 3.9 to be the culprit
I can access the containers from my local machine as before

I haven’t done anything other then saying "y" to create a bridge? in the lxd init process to networking at all, so this is all in complete default state

my LXD profile:

me@debian: lxc profile show default

config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: default
used_by:
- /1.0/containers/ubn1604
- /1.0/containers/ubn1604/sn1
- /1.0/containers/ubn1604/sn2
- /1.0/containers/ubn1604/sn3
[...]

my network interfaces

3: wlp2s0b1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether b0:10:41:b3:4f:c0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.178.137/24 brd 192.168.178.255 scope global dynamic wlp2s0b1
       valid_lft 862432sec preferred_lft 862432sec
    inet6 2003:ce:dbec:4800:87c4:2179:94d5:4cd5/64 scope global noprefixroute dynamic 
       valid_lft 7161sec preferred_lft 1761sec
    inet6 fe80::ef3e:9c26:9131:4309/64 scope link 
       valid_lft forever preferred_lft forever
[...]
8: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:43:eb:cf:30:5d brd ff:ff:ff:ff:ff:ff
    inet 10.19.225.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:2122:86a:2b30::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::687e:65ff:fef2:1509/64 scope link 
       valid_lft forever preferred_lft forever
[...]
16: vethF7TX51@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether fe:43:eb:cf:30:5d brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::fc43:ebff:fecf:305d/64 scope link 
       valid_lft forever preferred_lft forever

has anything changed in 3.9/3.8 in regards to networking?

stgraber · January 21, 2019, 8:40am

Hmm, nothing changed that would explain this, sounds like you’ve got a routing or firewalling issue going on here.

Can you show iptables -L -n -v and ip6tables -L -n -v?

vrms · January 21, 2019, 12:38pm

iptables -L -n -v

Chain INPUT (policy ACCEPT 454K packets, 601M bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53 /* generated for LXD network lxdbr0 */
   89  5792 ACCEPT     udp  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 /* generated for LXD network lxdbr0 */
   23  7544 ACCEPT     udp  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
    0     0 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:67

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    8   672 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    4   336 ACCEPT     all  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            /* generated for LXD network lxdbr0 */
    4   336 ACCEPT     all  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            /* generated for LXD network lxdbr0 */
    0     0 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker_gwbridge !docker_gwbridge  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      virbr0  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  virbr0 *       192.168.122.0/24     0.0.0.0/0           
    0     0 ACCEPT     all  --  virbr0 virbr0  0.0.0.0/0            0.0.0.0/0           
    0     0 REJECT     all  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 DROP       all  --  docker_gwbridge docker_gwbridge  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 315K packets, 153M bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            tcp spt:53 /* generated for LXD network lxdbr0 */
   89  6076 ACCEPT     udp  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            udp spt:53 /* generated for LXD network lxdbr0 */
   21  7056 ACCEPT     udp  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            udp spt:67 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            udp dpt:68

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker_gwbridge !docker_gwbridge  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    8   672 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

this looks a bit suspiscous

    0     0 REJECT     all  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

ip6tables -L -n -v (which I am not knowingly using)

Chain INPUT (policy ACCEPT 72398 packets, 73M bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp      lxdbr0 *       ::/0                 ::/0                 tcp dpt:53 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp      lxdbr0 *       ::/0                 ::/0                 udp dpt:53 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp      lxdbr0 *       ::/0                 ::/0                 udp dpt:547 /* generated for LXD network lxdbr0 */

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all      *      lxdbr0  ::/0                 ::/0                 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     all      lxdbr0 *       ::/0                 ::/0                 /* generated for LXD network lxdbr0 */

Chain OUTPUT (policy ACCEPT 53366 packets, 7829K bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp      *      lxdbr0  ::/0                 ::/0                 tcp spt:53 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp      *      lxdbr0  ::/0                 ::/0                 udp spt:53 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp      *      lxdbr0  ::/0                 ::/0                 udp spt:547 /* generated for LXD network lxdbr0 */

stgraber · January 21, 2019, 1:17pm

So I’m not actually noticing anything particularly wrong above, assuming your containers are all on lxdbr0 and not on virbr0 or docker0.

Can you also show iptables -t nat -L -n -v, maybe the problem is on the masquerading side?

Can you also show cat /proc/sys/net/ipv4/ip_forward for good measure?

vrms · January 21, 2019, 4:15pm

thans stephane,

Chain PREROUTING (policy ACCEPT 30087 packets, 8308K bytes)
 pkts bytes target     prot opt in     out     source               destination         
   69 11390 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 485 packets, 123K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 12768 packets, 817K bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 12763 packets, 816K bytes)
 pkts bytes target     prot opt in     out     source               destination         
    5   623 MASQUERADE  all  --  *      *       10.19.225.0/24      !10.19.225.0/24       /* generated for LXD network lxdbr0 */
    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           
    1    76 MASQUERADE  all  --  *      !docker_gwbridge  172.18.0.0/16        0.0.0.0/0           
    2   160 RETURN     all  --  *      *       192.168.122.0/24     224.0.0.0/24        
    0     0 RETURN     all  --  *      *       192.168.122.0/24     255.255.255.255     
    0     0 MASQUERADE  tcp  --  *      *       192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
    0     0 MASQUERADE  udp  --  *      *       192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
    0     0 MASQUERADE  all  --  *      *       192.168.122.0/24    !192.168.122.0/24    

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  docker_gwbridge *       0.0.0.0/0            0.0.0.0/0

furthermore …

@ debian  ~
└─ $ ▶ sudo cat /proc/sys/net/ipv4/ip_forward
1

all this does not really speak to me. And all containers are on the default profile and therfore go via the lxdbr0 interface

vrms · January 21, 2019, 7:06pm

I can ping 8.8.8.8, 8.8.4.4 from the containers, so it smells to me like a DNS problem of some sort.

EDIT: also I have noted that launching a new container (from an image I do not have locally) fails with some sort of connection problem

lxc launch ubuntu:18.04 ubn1804
Creating ubn1804
Error: Failed container creation: Get https://cloud-images.ubuntu.com/releases/streams/v1/index.json: lookup cloud-images.ubuntu.com on [::1]:53: read udp [::1]:48690->[::1]:53: read: connection refused

I can ping cloud-images.ubuntu.com from my host though.

vrms · January 22, 2019, 10:45am

did a sudo snap disable lxd & sudo snap enable lxd (which I had tried before without any effect) and all of a sudden the network problem seems to have disappeared.

thanks @stgraber for the efforts and sorry for the interruption.

vrms · January 23, 2019, 11:18am

today, after a restart of my host I see the same problem from withing containers.

snap disable lxd  
snap enable lxd

resolves this issue once again.

dellrusk · January 23, 2019, 11:38am

I think this may be related to the issue I reported recently : https://github.com/lxc/lxd-pkg-snap/issues/32

vrms · February 14, 2019, 10:34am

just wanted to update that this problem now is constant.

when my host (debian9 laptop) starts, I reach any internet from within any container (lxd and containers where autostarted on system boot).
I get reliably over this with a sudo snap disable lxd && sudo snap enable lxd command. I don’t believe this is as it is supposed to work.

pgoetz · February 14, 2019, 11:44am

So, I had this problem running LXD 3.9 on Arch Linux with an Ubuntu 14.04 container and reported in another ticket in General (Automatic name resolution within an Ubuntu 14.04 container?). The solution for this particular container was to provide it with an appropriate DNS server IP (in my case, 192.168.1.1):

$ lxc exec my_container bash
- Edit /etc/resolvconf/resolv.conf.d/base:

 nameserver  192.168.1.1

# resolvconf -u

What I couldn’t and still don’t understand is why this works automatically for an Ubuntu 16.04 host / Ubuntu 14.04 container, but not when the host machine is Arch Linux.

gpatel-fr · March 3, 2019, 8:16pm

For this kind of problem the first thing to do is check if dnsmasq is launched, if not look for syslog error messages to see why it fails to start, if yes check if it’s bound to the proper interface (by default lxdbr0)

zacksiri · March 8, 2019, 4:25am

I had this exact problem using ubuntu 18.04 I upgraded to 18.10 and everything worked again.

vrms · March 8, 2019, 1:40pm

thanks for sharing. In my case this is debian9 so there isn’t an option to upgrade to ubuntu 18.10. Neither do I believe everybody running an LTS ubuntu production server would want to upgrade to 18.10 necessarily.

So I still hope for a solution to come up. Till then I’ll live with my workaround (sudo snap disable lxd && sudo snap enable lxd)

gpatel-fr · March 8, 2019, 2:39pm

is there any difference in the output of
sudo ss -tapnu | grep LISTEN | grep 53
when it works and when it does not work ?

vrms · March 9, 2019, 9:40am

on an empty ubuntu:18.04 container, no

working

lxc exec ubn1804 -- sudo ss -tapnu | grep LISTEN | grep 53
tcp  LISTEN   0      128     127.0.0.53%lo:53          0.0.0.0:*       users:(("systemd-resolve",pid=112,fd=13))

not working

lxc exec ubn1804 -- sudo ss -tapnu | grep LISTEN | grep 53
tcp  LISTEN   0      128     127.0.0.53%lo:53          0.0.0.0:*       users:(("systemd-resolve",pid=112,fd=13))

gpatel-fr · March 9, 2019, 3:07pm

err, it was about launching ss on the host, not in a container, I was not precise enough.

vrms · March 10, 2019, 6:09am

on the debian9 host

1. while network inside container not working

(‘unknown host’, or ‘Temporary failure in name resolution’

$ ▶ sudo ss -tapnu | grep LISTEN | grep 53
[sudo] password for manolo: 
tcp    LISTEN     0      5      10.19.225.1:53                    *:*                   users:(("dnsmasq",pid=2709,fd=9))
tcp    LISTEN     0      5      192.168.122.1:53                    *:*                   users:(("dnsmasq",pid=2405,fd=6))
tcp    LISTEN     0      5         fd42:2122:86a:2b30::1:53                   :::*                   users:(("dnsmasq",pid=2709,fd=13))
tcp    LISTEN     0      5      fe80::74e3:bff:fee3:1197%lxdbr0:53                   :::*                   users:(("dnsmasq",pid=2709,fd=11))

2. After “sudo snap disable lxd && sudo snap enable lxd”

(so, ping is working inside containers)

─ $ ▶ sudo ss -tapnu | grep LISTEN | grep 53
tcp    LISTEN     0      5      10.19.225.1:53                    *:*                   users:(("dnsmasq",pid=9148,fd=9))
tcp    LISTEN     0      5      192.168.122.1:53                    *:*                   users:(("dnsmasq",pid=2405,fd=6))
tcp    LISTEN     0      128      :::80                   :::*                   users:(("apache2",pid=7356,fd=4),("apache2",pid=7355,fd=4),("apache2",pid=7354,fd=4),("apache2",pid=7353,fd=4),("apache2",pid=7352,fd=4),("apache2",pid=1625,fd=4))
tcp    LISTEN     0      5         fd42:2122:86a:2b30::1:53                   :::*                   users:(("dnsmasq",pid=9148,fd=13))
tcp    LISTEN     0      5      fe80::2ca4:aff:fe64:e46%lxdbr0:53                   :::*                   users:(("dnsmasq",pid=9148,fd=11))

gpatel-fr · March 11, 2019, 12:29am

well, nothing to see here. All seems normal when you have this problem.
What about
systemd-resolve --status in the container when you have the problem ?
and while I am at it,
ip addr
and
dig @10.19.225.1 ubuntu.com
the same, in the container with the problem.

vrms · March 11, 2019, 7:48am

Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 21 (eth0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 10.19.225.1
                      fd42:2122:86a:2b30::1
                      fe80::a49d:b2ff:fe77:45aa
          DNS Domain: lxd

does not tell me much to be honest

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
21: eth0@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:40:2c:9c brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.19.225.31/24 brd 10.19.225.255 scope global dynamic eth0
       valid_lft 2928sec preferred_lft 2928sec
    inet6 fd42:2122:86a:2b30:216:3eff:fe40:2c9c/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 3545sec preferred_lft 3545sec
    inet6 fe80::216:3eff:fe40:2c9c/64 scope link 
       valid_lft forever preferred_lft forever


; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @10.19.225.1 ubuntu.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 61342
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;ubuntu.com.			IN	A

;; Query time: 0 msec
;; SERVER: 10.19.225.1#53(10.19.225.1)
;; WHEN: Mon Mar 11 07:46:56 UTC 2019
;; MSG SIZE  rcvd: 28