Internet access issue inside container

Hello,

I have a little issue with Internet access inside a LXC container on a brand new server. Basically inside the container I can ping the host through the bridge but I cannot ping any external URL (e.g. 8.8.8.8). Also DNS name resolution works perfectly fine.

By using tcpdump I saw that the following output:

09:59:59.154470 IP 192.168.100.78 > 8.8.8.8: ICMP echo request, id 649, seq 0, length 64
10:00:00.155576 IP 192.168.100.78 > 8.8.8.8: ICMP echo request, id 649, seq 1, length 64

Has you can see the source IP is the address of the container not my public IP and I don’t know why.
I have the ip_forward flag enable (cat /proc/sys/net/ipv4/ip_forward -> 1) and I have the MASQUERADE rule created by lxc-net.

# Generated by xtables-save v1.8.3 on Thu Jul 18 10:36:10 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A POSTROUTING -s 192.168.100.0/24 ! -d 192.168.100.0/24 -j MASQUERADE
COMMIT
# Completed on Thu Jul 18 10:36:10 2019

So if you have any idea about how I can resolve this issue, please tell me.

PS: I’m using lxc version 3.0.3 on a Debian 9 (and container is Debian 10)

Which interface did you run tcpdump on?

I run it on the public interface of my server (eno1). Here is the full command line used: tcpdump -i eno1 -n icmp

Can you paste the output of

 iptables -L -v -n -t nat

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  *      *       192.168.100.0/24    !192.168.100.0/24    

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
# Warning: iptables-legacy tables present, use iptables-legacy to see them

@Forbinn thanks, all the packet counts are zero, have you recently reloaded them, can you try running your ping test again please.

Non I haven’t reloaded them. And even after running about 10 ping the result is still the same.

@Forbinn ok so thats pretty weird, seems like iptables isnt working correctly.

Can you show the output of iptables -L -v -n -t filter please.

Also, have you looked at the iptables-legacy comment at the bottom, could it be that this actual ruleset isnt being used?

iptables -L -v -n -t filter

Chain INPUT (policy ACCEPT 294K packets, 1082M bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  lxc-bridge-nat *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
    0     0 ACCEPT     udp  --  lxc-bridge-nat *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     tcp  --  lxc-bridge-nat *       0.0.0.0/0            0.0.0.0/0            tcp dpt:67
   68 22352 ACCEPT     udp  --  lxc-bridge-nat *       0.0.0.0/0            0.0.0.0/0            udp dpt:67
 237K 1079M f2b-recidive  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           
 227K 1047M f2b-sshd   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      lxc-bridge-nat  0.0.0.0/0            0.0.0.0/0           
   94  7896 ACCEPT     all  --  lxc-bridge-nat *       0.0.0.0/0            0.0.0.0/0           
    0     0 f2b-recidive  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 271K packets, 23M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain f2b-sshd (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 225K 1047M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain f2b-recidive (2 references)
 pkts bytes target     prot opt in     out     source               destination         
   51  3060 REJECT     all  --  *      *       51.77.254.207        0.0.0.0/0            reject-with icmp-port-unreachable
  874 52624 REJECT     all  --  *      *       106.75.71.124        0.0.0.0/0            reject-with icmp-port-unreachable
  246 14760 REJECT     all  --  *      *       51.75.23.87          0.0.0.0/0            reject-with icmp-port-unreachable
 236K 1079M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
# Warning: iptables-legacy tables present, use iptables-legacy to see them

For the iptables-legacy I have looked at them (both filter and nat) and there are all empty.
iptables-legacy -L -v -n -t filter

Chain INPUT (policy ACCEPT 296K packets, 1082M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 642 packets, 48504 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 271K packets, 23M bytes)
 pkts bytes target     prot opt in     out     source               destination

iptables-legacy -L -v -n -t nat

Chain PREROUTING (policy ACCEPT 61645 packets, 2302K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 59092 packets, 2146K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 3021 packets, 216K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 3173 packets, 226K bytes)
 pkts bytes target     prot opt in     out     source               destination

Ho, I’m misread your comment on iptables-legacy. After adding the MASQUERADE rule using the iptables-legacy and not the actual iptable the system works perfectly fine. So thank you for your help.

However why is this the case ? And how can I make this work using the non-legacy system ?

@Forbinn Great! :slight_smile:

Can I confirm that the original MASQUERADE rule was added by LXD and not yourself?

@tomp I confirm that the MASQUERADE rule is added by the lxc-net service. To be precise it is added by the script /usr/lib/x86_64-linux-gnu/lxc/lxc-net at line 110. BTW the problem also exists with IPv6 and can be resolved with the same technique.

@Forbinn OK so LXC is adding the rule using the correct iptables tool, which is then using iptables-nft , however your system still seems to be using the legacy iptables tables.

I would try moving all of your rules over to iptables-nft tables and then removing the old iptables-legacy tables (possibly unloading the modules too).

This might help: https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1688389.html

Sorry for the late reply, I had quite a few issues with the module unloading. But after doing so, it now works perfectly. So thanks again @tomp