What is the best way to create an internal network in lxc from client to server?


#1

I have been trying for 1 week and still can’t get client to communicate out via server to the internet (see my previous post for that).

I thought maybe there is a better lxc based way to do it as I was trying to follow along with a virtualbox example and adapt it for lxc.

I simply want a container acting as a server which accesses the internet with another internal network where a client container can communicate with the server in order to access the internet via ports 80/443 routed to port 8080 on the server.

How would that be done in lxc? I have been trying all week and can’t figure out how to setup the network devices so the client container can access the server container and get out to the internet/the specified port. As far as I have gotten so far is being able to ping the server’s gateway but not the internet (again see my recent post/s if you want further details on those failed attempts).

So perhaps taking a fresh look at how that would be done from scratch in lxc?


#2

In a nutshell, do you want to create containers than can only get Internet access through another special container (the default gateway)?


#3

@simos Yes that’s it.

Container 1 would be the one with internet access and I want the client container 2 to access internet only through the server on the specified port (8080). I am running mitmproxy on the server to intercept the traffic so I can analyze http requests from the client. Server is to act as transparent proxy.


#4

There are several networking options. @bmullan started a general discussion about this at LXD - new "white-box" networking technologies using LXD

@stgraber, can you give us a direction on a simple way to get a container to have to go through another container in order to reach the Internet?


#5

Oh I don’t want to have to use. Don’t see the point of that added layer.

I found this article which looks like it should do just what I want however I didn’t get it going yet.

I have now setup the br-cont0 device on brctl and added it to both server and client containers as a device giving them different IPs. I can ping back and forth but once again I am stuck figuring out how to route the traffic to port 8080/the outside internet.

Of course I am not following the whole thing but only the pertinent part of connecting two containers together.

Any ideas?

EDIT:

To update I did tcpdump -i <internal network> -vv on host and ping 8.8.8.8 on client; tcpdump showed output but client said host is unreachable and nothing showing in mitmproxy.


#6

You can then set up the gateway of one container to point to the other container.

The following is for LXD, and should work for LXC.

Create two containers, interceptor and victim.
Then, set the gateway for the victim to the IP address of the interceptor.
You can do that with manually configuring the network settings of the victim.
Keep the network settings that were provided by LXD and just change the gateway to the IP of the interceptor.

Go in the interceptor container and do
https://docs.mitmproxy.org/master/howto-transparent/

That should be it.


#7

That is exactly what I have been trying from the start unsuccessfully isn’t it? I don’t see anything different in that setup?


#8

I tried it before posting and it worked for me (on LXD 3.0).
If you want to try again, follow the instructions at https://docs.mitmproxy.org/master/howto-transparent/
and verify on each step that any commands are actually applied.
There might be an issue with what LXCFS version you are running.
I am not familiar with LXC to try there as well.


#9

Are you saying to use two network interfaces or just one for interceptor? If you meant just 1 then I tried it that way and didn’t work. There was no internet connection out.

I have tried for a whole week time and time again following the article exactly as it was written.


#10

I am using one interface for the interceptor.

Container victim

  1. lxc launch images:alpine/3.6 victim
  2. lxc exec victim -- /bin/sh
  3. apk update
  4. apk add w3m
  5. ifdown eth0
  6. # cat /etc/network/interfaces
    iface eth0 inet static
        address 10.52.252.52
        netmask 255.255.255.0
        gateway 10.52.252.252
        dns-nameserver 8.8.8.8 1.1.1.1
        hostname $(hostname)
    
  7. ifup eth0

The IP address is the IP address that was given by LXD’s dnsmasq. It’s nice to keep the same.
The gateway is the IP address of the interceptor container.
This container will not have Internet access unless IP Forwarding was enabled in the interceptor container.

Container interceptor:

  1. lxc launch ubuntu: interceptor
  2. lxc exec interceptor -- /bin/bash
  3. sysctl -w net.ipv4.ip_forward=1
    Now go to the victim container and w3m google.com. It should work.

If it does not, then there is an issue.

Btw, if you try tcpdump in containers, you may encounter an issue where you do not see any packets. In fact, the packets arrive but are cached. Just a note.


#11

No it doesn’t work. Just the same as my previous attempts.

NAME        STATE   AUTOSTART GROUPS IPV4         IPV6 
arch1       STOPPED 0         -      -            -    
mitm        RUNNING 0         -      xxx.xxx.3.1  -    
mitm-client RUNNING 0         -      xxx.xxx.3.10 -    
snap1       STOPPED 0         -      -            -    
snap2       STOPPED 0         -      -            -    
snap3       STOPPED 0         -      -            -

#interceptor and victim network config
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.name = eth0

interceptor network file
/etc/systemd/network/eth0.network
[Match]
Name=eth0

[Network]
Address=xxx.xxx.3.1/24
DNS=8.8.8.8
Note that I also tried with randomly generated ip that lxc gave me first and that time gave network unreachable.

vim network file
/etc/systemd/network/eth0.network
Address=xxx.xxx.3.10/24
Gateway=xxx.xxx.3.1
DNS=8.8.8.8

The biggest change I have gotten is following this article I was able to get some connectivity when using a different bridge. The ping would change the message to saying it was redirecting then next hop 8.8.8.8 then give an error so maybe that is progress. So I will continue trying this avenue of enquiry from the info in that article unless anyone offers a better idea.


#12

Update:

Yes! Finally did it!

I was missing the MASQUERADE line in ufw

-A POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE

Ok not quite done yet as mitm proxy isn’t working yet but I got internet with that line above so I guess I just have to tweak the iptables rules a bit more to get mitmproxy working. I know where to focus my attention now.

EDIT: Got it. Just needed some more ufw/iptables tweaks.


#14

So what tweaks did you do?

(Sorry, I’m a noob to the under the hood stuff and your perambulations help me learn the why of what’s going on.)

Please


#15

It was just routing the port forwardings and adding that masquerade line. I had tried one and not the other each time before, not both together.


#16

Thanks for the information!