Trouble setting up internal network on server/client containers


#1

I am having a nightmare getting these working. Been at it for days and can’t figure it out.

I get an error when I set the second interface via the config file for the server

lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.name = eth0
lxc.net.0.hwaddr = 00:16:3e:77:68:67 
lxc.net.0.ipv4.address = 10.0.xxx.xxx
lxc.net.0.ipv4.gateway = 10.0.3.1
#iface eth0 inet manual

lxc.net.2.type = veth
lxc.net.2.link = lxcbr0
lxc.net.2.flags = up
lxc.net.2.name = eth2
lxc.net.2.hwaddr = 00:16:3e:77:68:68
lxc.net.2.ipv4.address = 192.168.xxx.xxx
lxc.net.2.ipv4.gateway = 0.0.0.0

No matter what number I put for the second device it always says the error is that that ethx file exists even though I have never made one at that number before?

$  sudo lxc-start -n mitm -F
lxc-start: mitm: network.c: lxc_setup_netdev_in_child_namespaces: 2919 Failed to setup ipv4 gateway for network device "eth2": File exists
                                             lxc-start: mitm: network.c: lxc_setup_network_in_child_namespaces: 2991 failed to setup netdev
                                              lxc-start: mitm: conf.c: lxc_setup: 3158 failed to setup the network for 'mitm'
                                lxc-start: mitm: start.c: do_start: 944 Failed to setup container "mitm".
            lxc-start: mitm: sync.c: __sync_wait: 57 An error occurred in another process (expected sequence number 5)
                         lxc-start: mitm: start.c: __lxc_start: 1459 Failed to spawn container "mitm".
         lxc-start: mitm: tools/lxc_start.c: main: 371 The container failed to start.
lxc-start: mitm: tools/lxc_start.c: main: 375 Additional information can be obtained by setting the --logfile and --logpriority options.

Now I am able to successfully create the second one and assign it an IP by doing

lxc.net.2.type = veth
lxc.net.2.link = lxcbr0
lxc.net.2.flags = up
lxc.net.2.name = eth1
lxc.net.2.hwaddr = 00:16:3e:77:68:68

and then setting in /etc/systemd/network/eth1

[Match]
Name=eth1

[Network]
Address=xxx.xxx.3.1/24
Gateway=0.0.0.0

In this case when I start lxc it doesn’t produce the above error and assigns and ip but I still cannot communicate between the client container and the server.

The client I setup the network via the same way above in the container to assign the ip

[Match]
Name=eth1

[Network]
Address=xxx.xxx.3.10/24
Gateway=xxx.xxx.3.1
DNS=xxx.xxx.3.1

Following this but with containers.

I am not sure what I am missing.

I have setup static ip for the internal network on the server and set the client container to point to it but there is still no connection on the client.

Data for the containers

NAME        STATE   AUTOSTART GROUPS IPV4                   IPV6 
mitm        RUNNING 0         -      xxx.xxx.xxx.xxx (internet address for eth0 which works), xxx.xxx.3.1 (internal address for eth1) -    
mitm-client RUNNING 0         -      xxx.xxx.3.10           -    

Dnsmasq is running on the server.

$ sudo systemctl status dnsmasq
● dnsmasq.service - A lightweight DHCP and caching DNS server
   Loaded: loaded (/usr/lib/systemd/system/dnsmasq.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-04-10 15:58:21 UTC; 1min 23s ago
     Docs: man:dnsmasq(8)
  Process: 83 ExecStartPre=/usr/bin/dnsmasq --test (code=exited, status=0/SUCCESS)
 Main PID: 88 (dnsmasq)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/dnsmasq.service
           └─88 /usr/bin/dnsmasq -k --enable-dbus --user=dnsmasq --pid-file

Apr 10 15:58:21 mitm dnsmasq[83]: dnsmasq: syntax check OK.
Apr 10 15:58:21 mitm systemd[1]: Started A lightweight DHCP and caching DNS server.
Apr 10 15:58:21 mitm dnsmasq[88]: started, version 2.79 cachesize 150
Apr 10 15:58:21 mitm dnsmasq[88]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DH
Apr 10 15:58:21 mitm dnsmasq[88]: DBus support enabled: connected to system bus
Apr 10 15:58:21 mitm dnsmasq-dhcp[88]: DHCP, IP range xxx.xxx.x.xxx -- xxx.xxx.x.xxx, lease ti
Apr 10 15:58:21 mitm dnsmasq[88]: reading /etc/resolv.conf
Apr 10 15:58:21 mitm dnsmasq[88]: using nameserver 84.200.69.80#53
Apr 10 15:58:21 mitm dnsmasq[88]: using nameserver 84.200.70.40#53
Apr 10 15:58:21 mitm dnsmasq[88]: read /etc/hosts - 1 addresses

Route for server

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         xxx.xxx.3.1        0.0.0.0         UG    0      0        0 eth0
0.0.0.0         0.0.0.0         0.0.0.0         U     0      0        0 eth1
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 eth0
xxx.xxx.3.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1

Ip route on client

# ip route
default via xxx.xxx.3.1 dev eth0 proto static
xxx.xxx.3.0/24 dev eth0 proto kernel scope link src xxx.xxx.3.10

I start mitmproxy as per the link on the server with mitmproxy -T --host but there is still no internet connection on the client when I try ping or wget.

Any ideas what is wrong?