Internet access issue inside container

Forbinn · July 18, 2019, 8:37am

Hello,

I have a little issue with Internet access inside a LXC container on a brand new server. Basically inside the container I can ping the host through the bridge but I cannot ping any external URL (e.g. 8.8.8.8). Also DNS name resolution works perfectly fine.

By using tcpdump I saw that the following output:

09:59:59.154470 IP 192.168.100.78 > 8.8.8.8: ICMP echo request, id 649, seq 0, length 64
10:00:00.155576 IP 192.168.100.78 > 8.8.8.8: ICMP echo request, id 649, seq 1, length 64

Has you can see the source IP is the address of the container not my public IP and I don’t know why.
I have the ip_forward flag enable (cat /proc/sys/net/ipv4/ip_forward → 1) and I have the MASQUERADE rule created by lxc-net.

# Generated by xtables-save v1.8.3 on Thu Jul 18 10:36:10 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A POSTROUTING -s 192.168.100.0/24 ! -d 192.168.100.0/24 -j MASQUERADE
COMMIT
# Completed on Thu Jul 18 10:36:10 2019

So if you have any idea about how I can resolve this issue, please tell me.

PS: I’m using lxc version 3.0.3 on a Debian 9 (and container is Debian 10)

tomp · July 18, 2019, 11:20am

Which interface did you run tcpdump on?

Forbinn · July 18, 2019, 11:42am

I run it on the public interface of my server (eno1). Here is the full command line used: tcpdump -i eno1 -n icmp

tomp · July 18, 2019, 12:23pm

Can you paste the output of

 iptables -L -v -n -t nat

Forbinn · July 18, 2019, 12:25pm

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  *      *       192.168.100.0/24    !192.168.100.0/24    

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
# Warning: iptables-legacy tables present, use iptables-legacy to see them

tomp · July 18, 2019, 12:29pm

@Forbinn thanks, all the packet counts are zero, have you recently reloaded them, can you try running your ping test again please.

Forbinn · July 18, 2019, 12:31pm

Non I haven’t reloaded them. And even after running about 10 ping the result is still the same.

tomp · July 18, 2019, 12:33pm

@Forbinn ok so thats pretty weird, seems like iptables isnt working correctly.

Can you show the output of iptables -L -v -n -t filter please.

tomp · July 18, 2019, 12:34pm

Also, have you looked at the iptables-legacy comment at the bottom, could it be that this actual ruleset isnt being used?

Forbinn · July 18, 2019, 12:36pm

iptables -L -v -n -t filter

Chain INPUT (policy ACCEPT 294K packets, 1082M bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  lxc-bridge-nat *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
    0     0 ACCEPT     udp  --  lxc-bridge-nat *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     tcp  --  lxc-bridge-nat *       0.0.0.0/0            0.0.0.0/0            tcp dpt:67
   68 22352 ACCEPT     udp  --  lxc-bridge-nat *       0.0.0.0/0            0.0.0.0/0            udp dpt:67
 237K 1079M f2b-recidive  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           
 227K 1047M f2b-sshd   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      lxc-bridge-nat  0.0.0.0/0            0.0.0.0/0           
   94  7896 ACCEPT     all  --  lxc-bridge-nat *       0.0.0.0/0            0.0.0.0/0           
    0     0 f2b-recidive  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 271K packets, 23M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain f2b-sshd (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 225K 1047M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain f2b-recidive (2 references)
 pkts bytes target     prot opt in     out     source               destination         
   51  3060 REJECT     all  --  *      *       51.77.254.207        0.0.0.0/0            reject-with icmp-port-unreachable
  874 52624 REJECT     all  --  *      *       106.75.71.124        0.0.0.0/0            reject-with icmp-port-unreachable
  246 14760 REJECT     all  --  *      *       51.75.23.87          0.0.0.0/0            reject-with icmp-port-unreachable
 236K 1079M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
# Warning: iptables-legacy tables present, use iptables-legacy to see them

For the iptables-legacy I have looked at them (both filter and nat) and there are all empty.
iptables-legacy -L -v -n -t filter

Chain INPUT (policy ACCEPT 296K packets, 1082M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 642 packets, 48504 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 271K packets, 23M bytes)
 pkts bytes target     prot opt in     out     source               destination

iptables-legacy -L -v -n -t nat

Chain PREROUTING (policy ACCEPT 61645 packets, 2302K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 59092 packets, 2146K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 3021 packets, 216K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 3173 packets, 226K bytes)
 pkts bytes target     prot opt in     out     source               destination

Forbinn · July 18, 2019, 1:04pm

Ho, I’m misread your comment on iptables-legacy. After adding the MASQUERADE rule using the iptables-legacy and not the actual iptable the system works perfectly fine. So thank you for your help.

However why is this the case ? And how can I make this work using the non-legacy system ?

tomp · July 18, 2019, 3:50pm

@Forbinn Great!

Can I confirm that the original MASQUERADE rule was added by LXD and not yourself?

Forbinn · July 18, 2019, 4:31pm

@tomp I confirm that the MASQUERADE rule is added by the lxc-net service. To be precise it is added by the script /usr/lib/x86_64-linux-gnu/lxc/lxc-net at line 110. BTW the problem also exists with IPv6 and can be resolved with the same technique.

tomp · July 18, 2019, 4:54pm

@Forbinn OK so LXC is adding the rule using the correct iptables tool, which is then using iptables-nft , however your system still seems to be using the legacy iptables tables.

I would try moving all of your rules over to iptables-nft tables and then removing the old iptables-legacy tables (possibly unloading the modules too).

tomp · July 18, 2019, 5:00pm

This might help: Bug#932196: Load legacy iptables module 'iptable_filter' on nftables system

Forbinn · July 20, 2019, 8:05am

Sorry for the late reply, I had quite a few issues with the module unloading. But after doing so, it now works perfectly. So thanks again @tomp

r1ckdu · June 12, 2023, 7:18pm

Maybe not the right place but I searched and found this. I also have the same issue - dns lookup works but i cannot ping anything outside.

The container uses a network called lxdbr0.

My container is setup to work in an ubuntu environment and has docker also installed.

What i can see in iptables with :

$ sudo  iptables -L -v -n -t nat
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
52376 3420K DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
26076 2634K DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
   31  2014 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  all  --  *      !br-e3bc089bb3d9  172.19.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  all  --  *      !br-0b908a7dd82f  172.20.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  all  --  *      !br-05a0c150a474  172.18.0.0/16        0.0.0.0/0

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  br-e3bc089bb3d9 *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  br-0b908a7dd82f *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  br-05a0c150a474 *       0.0.0.0/0            0.0.0.0/0

And iptables-legacy:

$ sudo iptables-legacy -L -v -n -t nat
Chain PREROUTING (policy ACCEPT 2718 packets, 302K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 1146 packets, 220K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 1567 packets, 145K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 1557 packets, 141K bytes)
 pkts bytes target     prot opt in     out     source               destination

Interestingly lxdbr0 does not appear anywhere?

In lxc the network is setup like this:

$ lxc network show lxdbr0
config:
  ipv4.address: 10.187.231.1/24
  ipv4.firewall: "false"
  ipv4.nat: "true"
  ipv6.address: none
  ipv6.nat: "true"
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/ncp-backup
- /1.0/profiles/default
managed: true
status: Created
locations:
- none

What do i need to do to get this to work? I did not find the command @Forbinn used to add MASQUERADE rule to his things… would be real nice to have some input on this! Also maybe @tomp might have an idea?

Thanks y’all!!!

r1ckdu · June 12, 2023, 7:40pm

Maybe this helps - i looked into this other issue here

and ran the grep suggestion

$ ps aux | grep dnsmasq
lxd       452486  0.0  0.0   7208  3632 ?        Ss   18:34   0:00 dnsmasq --keep-in-foreground --strict-order --bind-interfaces --except-interface=lo --pid-file= --no-ping --interface=lxdbr0 --dhcp-rapid-commit --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=10.187.231.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases --dhcp-hostsfile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts --dhcp-range 10.187.231.2,10.187.231.254,1h -s lxd --interface-name _gateway.lxd,lxdbr0 -S /lxd/ --conf-file=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.raw -u lxd -g lxd

The container has an ip and I can ping the host internally:
ping 10.187.231.1 takes like 0.01ms
also for ping 192.168.1.98 (external host ip)
pinging the router 192.168.1.1 of course does not work…

tomp · June 13, 2023, 7:35am

As you have Docker running also checkout:

https://linuxcontainers.org/lxd/docs/master/howto/network_bridge_firewalld/#prevent-connectivity-issues-with-lxd-and-docker