Connection to apache server in container refused

LXD
1

I have an lxd container installed on my SSDNodes VPS (ip address=209.182.233.167)

I installed apache2 on the server. It was apparently successful because /var/www/html/index.html exists.

lxc list shows the containers ip address as 10.23.133.117.

As root in the VPS host (not in the container) I ran:
iptables -t nat -I PREROUTING -i eth0 -p TCP -d 209.182.233.167/32 --dport 80 -j DNAT --to-destination 10.23.133.117:80

When I enter 209.182.233.167 in a browser, I get a “connection refused” error.

What am I missing?

2

Hi,

Have you checked Apache is actually running inside the container yet? If not try, running inside the container:

curl http://127.0.0.1

If that works OK and you get an Apache welcome page, then try the same command on the host but with the container’s private IP, e.g.

curl http://10.23.133.117

If that works, then its likely an issue with your iptables rule, but first you should check you can ping the private IP from the host.

3

Thanks, tomp.
As root of the host server, I had run curl 10.23.133.117 before I read your post and got the html code for the index page.

As you suggested, I ran curl http://127.0.0.1. I got curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused

So inside the container I ran systemctl status apache2 and got:
root@inglesnobrasil:~# systemctl status apache2
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Drop-In: /lib/systemd/system/apache2.service.d
└─apache2-systemd.conf
Active: active (running) since Thu 2019-08-08 12:38:08 UTC; 1h 38min ago
Main PID: 3271 (apache2)
Tasks: 54 (limit: 4915)
CGroup: /system.slice/apache2.service
├─3271 /usr/sbin/apache2 -k start
├─3273 /usr/sbin/apache2 -k start
└─3274 /usr/sbin/apache2 -k start

Aug 08 12:38:08 inglesnobrasil systemd[1]: Starting The Apache HTTP Server...
Aug 08 12:38:08 inglesnobrasil systemd[1]: Started The Apache HTTP Server.
Aug 08 12:38:08 inglesnobrasil systemd[1]: apache2.service: Failed to reset devices.list: Operation not permitted
Aug 08 12:38:09 inglesnobrasil systemd[1]: apache2.service: Failed to reset devices.list: Operation not permitted
Aug 08 12:38:09 inglesnobrasil systemd[1]: apache2.service: Failed to reset devices.list: Operation not permitted
Aug 08 12:38:09 inglesnobrasil systemd[1]: apache2.service: Failed to reset devices.list: Operation not permitted

The error.log in /var/log/apache2 says:
[Thu Aug 08 12:38:08.764231 2019] [mpm_event:notice] [pid 3271:tid 140321393314752] AH00489: Apache/2.4.29 (Ubuntu) configure$
[Thu Aug 08 12:38:08.764621 2019] [core:notice] [pid 3271:tid 140321393314752] AH00094: Command line: ‘/usr/sbin/apache2’

4

if you can connect from the host, it does not matter than Apache don’t listen on the container localhost. So the problem is probably firewall related. BTW if you use Ubuntu you could use ufw for firewall configuration it’s easier (although not by much for anything related to masquerade or dnat).
I’d say that routing looks the main possible culprit.
If you want to learn how to debug such stuff yourself, use tcpdump.
On the host:
sudo tcpdump -i eth0 port 80
then do a curl from outside the network. If you see a new line for your curl in the tcpdump output you sent successfully your packet to your VPN. Somewhat likely but when debugging network connectivity, never assume anything.
Then if you get a correct result, still on the host:
sudo tcpdump -i lxdbr0 port 80
Then do a curl from outside the network. If you see a new line for your curl in the tcpdump output the host routes correctly the packet to your container network.
1: if it works: time to go into the container to launch tcpdump
2: it it don’t work - as I suspect - try to check the routing.
If ufw is enabled:
sudo ufw status verbose --> will show you if default routing is enabled or not
(deny:routed)
if routing is disallowed by default (an option I prefer myself) you may have to either allow it globally or create a specific rule such as

sudo ufw route allow in on eth0 out on lxdbr0 proto tcp to 10.23.133.117 port 80 comment "input from internet to container Apache"

Note that the lxd project has also added a proxy functionality to the system to manage such needs from inside lxd configuration, it’s possible to use it instead of iptables/ufw.

5

Sorry I didn’t reply sooner. I tried to, but I got a message that I’d already reached my daily limit of 4 replies and I had to wait.

When you say “on the host” I assume you mean the VPS which is running LXD. (My VPS is root@server1, inside the container I am root@inglesnobrasil.) When I ran tcpdump, I got:
root@server1:/home/randall# sudo tcpdump -i eth0 port 80
tcpdump: eth0: No such device exists

6

something is troubling here how can you set an iptables rules on a device that don’t exist ? I had assumed that your VPS network interface name was eth0.

7

Again I had to wait to reply.
Below is the result of running ifconfig in the host. Ehere it has ens3, shouldn’t it be eth0?

randall@server1:~$ ifconfig
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 209.182.233.167  netmask 255.255.255.0  broadcast 209.182.233.255
        inet6 fe80::21c:42ff:fee7:abc1  prefixlen 64  scopeid 0x20<link>
        ether 00:1c:42:e7:ab:c1  txqueuelen 1000  (Ethernet)
        RX packets 1405876  bytes 434459336 (434.4 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 48637  bytes 4925100 (4.9 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 7  bytes 376 (376.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7  bytes 376 (376.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lxcbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 10.0.3.1  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 00:16:3e:00:00:00  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lxdbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.23.133.1  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fd42:27bd:c49b:81a3::1  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::20ea:c9ff:fe11:4797  prefixlen 64  scopeid 0x20<link>
        ether fe:64:00:7b:42:87  txqueuelen 1000  (Ethernet)
        RX packets 2779  bytes 200211 (200.2 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4270  bytes 25034515 (25.0 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethTLI0R6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fc64:ff:fe7b:4287  prefixlen 64  scopeid 0x20<link>
        ether fe:64:00:7b:42:87  txqueuelen 1000  (Ethernet)
        RX packets 2779  bytes 239117 (239.1 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4280  bytes 25034619 (25.0 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
8

Well ens3 is the name the system has picked for your public interface and that just what it is.
So replace eth0 by ens3 in my previous post. I just checked and iptables does accept any interface name so your rule was basically a no operation.

9

I ran sudo iptables -t nat -I PREROUTING -i ens3 -p TCP -d 209.182.233.167/32 --dport 80 -j DNAT --to-destination 10.23.133.117:80 (Notice I used ens3 where I had previously used eth0.) and now it works.

I don’t get the significance of ens3 vs eth0.

While I was waiting to be able to reply, I read a few articles which seem to indicate ens3 is replacing eth0. I don’t know how this will be significant, bu at least now I know I need to get that info (eth0 vs ens3) from ifconfig to use in the iptable rule.

10

For your information, about any type of combination of computer/linux OS on the planet seems to have a specific name for the network interface. On my workstation it’s enp3s0; I have another computer with ens160, another server with ens192. I have never tried to find out why, I just live with it.
It this annoys you it’s possible to (re)name the interfaces at your liking using udev. I guess that’s was done for some of my systems where the thing is still named eth0.

11

Glad that worked @rpallen

iptables does allow you to create rules even if the interface doesn’t exist (this can actually be quite useful for testing loading iptables rules on a non-production system).

Linux will pick a name for each of the network interfaces in your machine or VPS. Modern systems use the ensXXX type naming and older systems use eth0 (although you can return modern systems to use eth0 if you prefer).

But for the scope of this thread, its important that iptables rules match the name of the actual interface on your machine/VPS as iptables will let you add rules for interfaces that don’t exist, and as in your case, the rule will just silently not work.

Where did you get eth0 from for your original rule by the way?

12

Thanks for your help.
No, it doesn’t annoy me. I don’t care what the interface is named as long as I can get my containers to work-and this one now works.
Thanks again

13

I copied the iptable rule from a tutorial and changed the ip addresses as appropriate. I thought the eth0 was a pretty standard thing that wouldn’t change. Now I know different.