Access container service from other machine

,
1

At home, I have a 192.168.10.0/24 subnet served by the wifi router.

I have three networking devices here, a server and a laptop connected via router.

Laptop (192.168.10.100) <----- Router 192.168.10.1 -------> Server (192.168.10.2)

Router is Tmobile AC1900 (ASUS router with Merlin firmware)
Laptop - Running Ubuntu 18.04.2 Desktop
Headless Server - Running Ubuntu 18.04.2 Server

On server I installed the LXD via snap (I believe the version is 3.10).

Fox the LXD init , I used all the default settings, excepts I used 192.168.5.0/24 as IPv4 subnet and disabled IPv6.

Now I created Ubuntu Xenial container to install pihole (I know I can use Docker or there are other ways, but that is not why I am asking the question) with the static IP of 192.168.5.5.

I added a static route to my router as follow

Network: 192.168.5.0
Subnetmask: 255.255.255.0
Gateway: 192.168.10.2 (Server IP)

With that settings, I can ping and tracepath the container IP 192.168.5.5.
However when it comes to accessing the pihole website, I can’t access it from the my laptop.

From the server curl 192.168.5.5 produces the html output as expected. However same call doesn’t go anywhere from the laptop.

I feel like I am very close to have things running as expected, but trying to wrap my head around networking and why HTTP is not accessible from other machine?

Who is answering the ping/tracepath calls with the container ip?

How do I achieve accessing the container service outside of the server using the container ip only.
I am trying to avoid port binding to the host because I plan to run http services on a different containers e.g nginx on 192.168.5.6 etc…

Any help or insight is appreciated related to container networking and how it works.
I did read redhat documentations, but it is just a wall of text and hard to comprehend.

 ping 192.168.5.5
PING 192.168.5.5 (192.168.5.5) 56(84) bytes of data.
64 bytes from 192.168.5.5: icmp_seq=1 ttl=63 time=4.11 ms
From 192.168.10.1: icmp_seq=2 Redirect Host(New nexthop: 192.168.10.2)
64 bytes from 192.168.5.5: icmp_seq=2 ttl=63 time=4.29 ms
From 192.168.10.1: icmp_seq=3 Redirect Host(New nexthop: 192.168.10.2)
64 bytes from 192.168.5.5: icmp_seq=3 ttl=63 time=3.36 ms
From 192.168.10.1: icmp_seq=4 Redirect Host(New nexthop: 192.168.10.2)
64 bytes from 192.168.5.5: icmp_seq=4 ttl=63 time=3.20 ms
From 192.168.10.1: icmp_seq=5 Redirect Host(New nexthop: 192.168.10.2)
64 bytes from 192.168.5.5: icmp_seq=5 ttl=63 time=4.79 ms
From 192.168.10.1: icmp_seq=6 Redirect Host(New nexthop: 192.168.10.2)
64 bytes from 192.168.5.5: icmp_seq=6 ttl=63 time=8.80 ms

 tracepath 192.168.5.5
 1?: [LOCALHOST]                      pmtu 1500
 1:  cellspot.router                                       3.840ms 
 1:  cellspot.router                                       1.635ms 
 2:  192.168.10.2                                          1.534ms asymm  1 
 3:  192.168.5.5                                           1.851ms reached
     Resume: pmtu 1500 hops 3 back 2
2

well, it should be your container. My guess is that your container default route is the server interface, the server default route is your router, so the reply from the container to your laptop goes to the router and then well the router know where to find your laptop since it’s on one of the 2 networks it manages.

Now why http don’t work while ping works, my guess is that some sort of firewall is adding joy to your life.
tcpdump is the network manager’s friend. I’d start from the server, tcpdump on the lxdbr0 interface
sudo tcpdump -i lxdbr0 host 192.168.10.100 then try a curl and see what happens. If the container sends a reply the problem is not with the container. And so on.

3

Thank you for the useful information. I will try it out and post my findings.

4

I captured some tcpdump on lxdbr0 and also on the primary interface for the server enp3s0.
In both instances I saw traffic coming to the interface. However at the receiving end I didn’t see any traffic.

I also created another container with just nginx installed and it was the same result.
I also tried web service running on different port, but same result. I see traffic on the tcpdump but on the client side I don’t see any traffic coming back.

sudo tcpdump -n  -i lxdbr0 host 192.168.10.228
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lxdbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:40:20.867097 IP 192.168.10.228.33324 > 192.168.5.5.80: Flags [S], seq 3777363264, win 29200, options [mss 1460,sackOK,TS val 3427772296 ecr 0,nop,wscale 7], length 0
21:40:20.867244 IP 192.168.5.5.80 > 192.168.10.228.33324: Flags [S.], seq 4119378268, ack 3777363265, win 28960, options [mss 1460,sackOK,TS val 3322019503 ecr 3427772296,nop,wscale 7], length 0
21:40:21.877805 IP 192.168.5.5.80 > 192.168.10.228.33324: Flags [S.], seq 4119378268, ack 3777363265, win 28960, options [mss 1460,sackOK,TS val 3322020514 ecr 3427772296,nop,wscale 7], length 0
21:40:23.893812 IP 192.168.5.5.80 > 192.168.10.228.33324: Flags [S.], seq 4119378268, ack 3777363265, win 28960, options [mss 1460,sackOK,TS val 3322022530 ecr 3427772296,nop,wscale 7], length 0
21:40:28.021807 IP 192.168.5.5.80 > 192.168.10.228.33324: Flags [S.], seq 4119378268, ack 3777363265, win 28960, options [mss 1460,sackOK,TS val 3322026659 ecr 3427772296,nop,wscale 7], length 0
21:40:36.213806 IP 192.168.5.5.80 > 192.168.10.228.33324: Flags [S.], seq 4119378268, ack 3777363265, win 28960, options [mss 1460,sackOK,TS val 3322034853 ecr 3427772296,nop,wscale 7], length 0
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel

After that I started tcpdump on the laptop as well. I do see some response coming back, but it never produces the output for the curl call.

➜  ~ sudo tcpdump -n -i wlp2s0 host 192.168.5.5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wlp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:18:09.050282 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [S], seq 49760554, win 29200, options [mss 1460,sackOK,TS val 3430040491 ecr 0,nop,wscale 7], length 0
22:18:09.055787 IP 192.168.5.5.80 > 192.168.10.228.48864: Flags [S.], seq 101089379, ack 49760555, win 28960, options [mss 1460,sackOK,TS val 3324288249 ecr 3430040491,nop,wscale 7], length 0
22:18:09.055878 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 3430040497 ecr 3324288249], length 0
22:18:09.056028 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [P.], seq 1:76, ack 1, win 229, options [nop,nop,TS val 3430040497 ecr 3324288249], length 75: HTTP: GET / HTTP/1.1
22:18:09.266956 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [P.], seq 1:76, ack 1, win 229, options [nop,nop,TS val 3430040708 ecr 3324288249], length 75: HTTP: GET / HTTP/1.1
22:18:09.478912 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [P.], seq 1:76, ack 1, win 229, options [nop,nop,TS val 3430040920 ecr 3324288249], length 75: HTTP: GET / HTTP/1.1
22:18:09.922953 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [P.], seq 1:76, ack 1, win 229, options [nop,nop,TS val 3430041364 ecr 3324288249], length 75: HTTP: GET / HTTP/1.1
22:18:10.155189 IP 192.168.5.5.80 > 192.168.10.228.48864: Flags [S.], seq 101089379, ack 49760555, win 28960, options [mss 1460,sackOK,TS val 3324289255 ecr 3430040491,nop,wscale 7], length 0
22:18:10.155253 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 3430041596 ecr 3324288249], length 0
22:18:10.786942 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [P.], seq 1:76, ack 1, win 229, options [nop,nop,TS val 3430042228 ecr 3324288249], length 75: HTTP: GET / HTTP/1.1
22:18:12.307837 IP 192.168.5.5.80 > 192.168.10.228.48864: Flags [S.], seq 101089379, ack 49760555, win 28960, options [mss 1460,sackOK,TS val 3324291272 ecr 3430040491,nop,wscale 7], length 0
22:18:12.307900 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 3430043749 ecr 3324288249], length 0
22:18:12.482798 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [P.], seq 1:76, ack 1, win 229, options [nop,nop,TS val 3430043924 ecr 3324288249], length 75: HTTP: GET / HTTP/1.1
22:18:15.842904 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [P.], seq 1:76, ack 1, win 229, options [nop,nop,TS val 3430047284 ecr 3324288249], length 75: HTTP: GET / HTTP/1.1
22:18:16.264477 IP 192.168.5.5.80 > 192.168.10.228.48864: Flags [S.], seq 101089379, ack 49760555, win 28960, options [mss 1460,sackOK,TS val 3324295369 ecr 3430040491,nop,wscale 7], length 0
22:18:16.264561 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 3430047706 ecr 3324288249], length 0
22:18:22.754759 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [P.], seq 1:76, ack 1, win 229, options [nop,nop,TS val 3430054196 ecr 3324288249], length 75: HTTP: GET / HTTP/1.1
22:18:24.362695 IP 192.168.5.5.80 > 192.168.10.228.48864: Flags [S.], seq 101089379, ack 49760555, win 28960, options [mss 1460,sackOK,TS val 3324303563 ecr 3430040491,nop,wscale 7], length 0
22:18:24.362799 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 3430055804 ecr 3324288249], length 0
22:18:27.865835 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [F.], seq 76, ack 1, win 229, options [nop,nop,TS val 3430059307 ecr 3324288249], length 0

Also curl in verbose mode

➜  ~ curl -v  192.168.5.5
* Rebuilt URL to: 192.168.5.5/
*   Trying 192.168.5.5...
* TCP_NODELAY set
* Connected to 192.168.5.5 (192.168.5.5) port 80 (#0)
> GET / HTTP/1.1
> Host: 192.168.5.5
> User-Agent: curl/7.58.0
> Accept: */*
>

Any more insight?

What should be lxdbr0 settings should look time ?

sudo lxc network show lxdbr0 
config:
  ipv4.address: 192.168.5.1/24
  ipv4.nat: "false"
  ipv6.address: none
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/containers/pihole
managed: true
status: Created
locations:
- none
5

hmm, it’s baffling indeed.
This should not happen :slight_smile:
The initial SYN packet goes in without problem.
The reply (SYN-ACK) goes in the opposite direction and your client sees it, it’s sending the second ACK but the server don’t see it (you can see in the first trace that it’s trying to send the first ACK again several times)
On the client side, you can see that it sees the first ACK and it is sending the second, so from the client point of view the connection succeeds; that’s what curl is saying.
Then the client tries to send application data (the HTTP GET packets) but the server don’t see it, not a single one.
Yes it looks like a firewall problem but not an obvious one (a missing rule), it’s at the kernel level I guess.
Probably a martian problem. Try to enable sysctl log_martian on all computers and look at the syslogs
(edit /etc/sysctl.conf and add net.ipv4.conf.all.log_martians=1 and net.ipv4.conf.default.log_martians=1, reload with sysctl -p)

About your configuration I almost never see this kind of problem probably because I always nat the lxdbr0 network

6

Again, thank you for your help.
I have lxdbr0 also running with NAT set to true. In both cases the behavior was the same.
I will try enabling martian logging and see what happens.
BTW I tried to access the http service from other devices (phone, tablet) besides the laptop, and it was the same result. So I suspect this is more related to server side.

7

If it’s a martian problem, normally an inverse route added to the server configuration should solve it.

8

I added logging but I am not sure which log to look at. There are not changes to any log files under /var/log after enabling the reloading the sysctl.
I tried several curl calls the target container after enabling the martian logging.
BTW, I am running LXD as a snap package, just additional information.

9

these messages are kernel messages so they should appear in whatever place kernel messages are logged. In my distro (Ubuntu) it’s syslog.
test with
sudo sysctl -a | grep martians
in syslog I can see messages like
…kernel: [391119.150842] IPv4: martian source …

you can try also to use tcpdump on the server interface (not lxdbr0, but the external host interface, eth0 or something else) to check if the third packet (the ACK of the SYN-ACK, in the same direction as the initial SYN) is getting to it.

you can use telnet 80 to simulate a connection, it’s less verbose than Curl, just to make tcpdump trace a bit more clear.

10 
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
net.ipv4.conf.docker0.log_martians = 0
net.ipv4.conf.enp3s0.log_martians = 0
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.lxdbr0.log_martians = 0
net.ipv4.conf.vethI8U99C.log_martians = 1
net.ipv4.conf.wlp2s0.log_martians = 1

Nothing for the syslog output
$ sudo grep martian /var/log/syslog

I also tailed the syslog when I used curl or telent in both cases there were not log messages in syslog.

I also captured the tcpdump on both interfaces and here is the output. I am not a networking expert so I am not exactly sure what to make out of it. (I have started reading on the TCP)

$ sudo tcpdump -n -i enp3s0 port 80    
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:49:23.315724 IP 192.168.10.228.40890 > 192.168.5.5.80: Flags [S], seq 998555094, win 29200, options [mss 1460,sackOK,TS val 3478714821 ecr 0,nop,wscale 7], length 0
11:49:23.316474 IP 192.168.5.5.80 > 192.168.10.228.40890: Flags [S.], seq 487295767, ack 998555095, win 28960, options [mss 1460,sackOK,TS val 3372974342 ecr 3478714821,nop,wscale 7], length 0
11:49:24.086043 IP 192.168.5.5.80 > 192.168.10.228.40882: Flags [S.], seq 1491401628, ack 2998704734, win 28960, options [mss 1460,sackOK,TS val 3372975111 ecr 3478684201,nop,wscale 7], length 0
11:49:24.341909 IP 192.168.5.5.80 > 192.168.10.228.40890: Flags [S.], seq 487295767, ack 998555095, win 28960, options [mss 1460,sackOK,TS val 3372975367 ecr 3478714821,nop,wscale 7], length 0
11:49:26.357936 IP 192.168.5.5.80 > 192.168.10.228.40890: Flags [S.], seq 487295767, ack 998555095, win 28960, options [mss 1460,sackOK,TS val 3372977384 ecr 3478714821,nop,wscale 7], length 0
11:49:30.485951 IP 192.168.5.5.80 > 192.168.10.228.40890: Flags [S.], seq 487295767, ack 998555095, win 28960, options [mss 1460,sackOK,TS val 3372981513 ecr 3478714821,nop,wscale 7], length 0
11:49:38.678038 IP 192.168.5.5.80 > 192.168.10.228.40890: Flags [S.], seq 487295767, ack 998555095, win 28960, options [mss 1460,sackOK,TS val 3372989707 ecr 3478714821,nop,wscale 7], length 0
^C
7 packets captured
7 packets received by filter
0 packets dropped by kernel

$ sudo tcpdump -n -i lxdbr0 port 80 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lxdbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:49:51.477576 IP 192.168.10.228.40898 > 192.168.5.5.80: Flags [S], seq 1213562173, win 29200, options [mss 1460,sackOK,TS val 3478742980 ecr 0,nop,wscale 7], length 0
11:49:51.478207 IP 192.168.5.5.80 > 192.168.10.228.40898: Flags [S.], seq 623557284, ack 1213562174, win 28960, options [mss 1460,sackOK,TS val 3373002510 ecr 3478742980,nop,wscale 7], length 0
11:49:52.501808 IP 192.168.5.5.80 > 192.168.10.228.40898: Flags [S.], seq 623557284, ack 1213562174, win 28960, options [mss 1460,sackOK,TS val 3373003534 ecr 3478742980,nop,wscale 7], length 0
11:49:54.517807 IP 192.168.5.5.80 > 192.168.10.228.40898: Flags [S.], seq 623557284, ack 1213562174, win 28960, options [mss 1460,sackOK,TS val 3373005551 ecr 3478742980,nop,wscale 7], length 0
11:49:54.805824 IP 192.168.5.5.80 > 192.168.10.228.40890: Flags [S.], seq 487295767, ack 998555095, win 28960, options [mss 1460,sackOK,TS val 3373005839 ecr 3478714821,nop,wscale 7], length 0
11:49:58.645799 IP 192.168.5.5.80 > 192.168.10.228.40898: Flags [S.], seq 623557284, ack 1213562174, win 28960, options [mss 1460,sackOK,TS val 3373009680 ecr 3478742980,nop,wscale 7], length 0
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel
11

BTW this solution works :blush: I don’t fully understand the difference between macvlan and bridged mode. LXD needs better documentation. I think Docker beats LXD to that, which probably also attributes to its success. :man_shrugging:

12

The problem does not seem to be on your server, since on the interface the second ACK is nowhere to be seen. As this ACK (the third packet) can be seen exiting out of your client (laptop) network interface:

22:18:09.050282 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [S], seq 49760554, win 29200, options [mss 1460,sackOK,TS val 3430040491 ecr 0,nop,wscale 7], length 0
22:18:09.055787 IP 192.168.5.5.80 > 192.168.10.228.48864: Flags [S.], seq 101089379, ack 49760555, win 28960, options [mss 1460,sackOK,TS val 3324288249 ecr 3430040491,nop,wscale 7], length 0
22:18:09.055878 IP 192.168.10.228.48864 > 192.168.5.5.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 3430040497 ecr 3324288249], length 0

see the third line ? That’s the packet that is not getting to the network interface of your server.
So it is probably the thing called ‘cellspot router’ in your traceroute that is blocking this packet.

if you use macvlan, your container does get an address directly from the network so there is no routing,

thinking back I realize now that there is something very wrong in your setup.
It’s so big that somehow I missed it. You setup a route in your router. Well, there is no router. A router connects 2 different networks. There is only ONE network for your client and your server. Your router is really a bridge between them. That should be why it is confused with the ‘route’. What you really need is to setup your ‘router’ to distribute the route to the clients (if your router is doing DHCP that is, if not the only way is to setup the route manually in each client, that is, the laptop)

13

Thank you very much… Care to elaborate in the route distribution part from router?
It is an off the shelve ASUS router with Merlin firmware. I didn’t see any part on the router configuration that enabled the route distribution or route learning.

14

That’s a DHCP feature in fact. When a DHCP server gives an address to a client computer, it can also add general network configuration, usually a DNS server address and a default gateway, but more is possible, see
https://tools.ietf.org/html/rfc3442
however, all depends on the dhcp software. AFAIK the Linux dhcp client supports it, the classic dhcp server (from the ISC) supports it, dnsmasq don’t seem to support it. I have no idea about dd-wrt, sorry.
For a 2 computers network it seems a bit of an overkill anyway and your could just drop the route in your ‘cell router’ and add it by hand on the laptop.

15

Thank you… I have more devices in the network. But I was just testing it.
I think I can live with macvlan for the pihole solution, which then can act as DHCP server and can push routes to client. I still have to test it out how it will affect other non Linux clients.
Thanks for all the help and details you provided, really appreciate it.