Problem using MACVLAN in NATted network

networking

(Johannes Midgren) #1

My issue is that I’m only able to reach a container with two network devices from Internet using port forwarding on the router if I use a specific order of the defined network devices. The macvlan device has to come first.

Background

I would like to set up a number of web services using LXD. My idea is to use one container as an HTTP reverse proxy (defining several vhosts in apache and handle all SSL) and then install various services such as NextCloud, MythWeb and ZoneMinder in separate containers. Communication between the containers could use the default lxdbr0 and the reverse proxy should communicate with the outside world using macvlan.

After reading a few topics here, and especially the following two great replies by @stgraber, I thought I understood a bit more about how a macvlan setup works:

Configuration

I use the following profile to create the basics of the reverse proxy container:

config:
  environment.http_proxy: ""
  user.network-config: |
    version: 1
    config:
      - type: physical
        name: eth0
        subnets:
          - type: dhcp
            ipv4: true
            control: auto
      - type: physical
        name: eth1
        subnets:
          - type: dhcp
            ipv4: true
            control: auto
  user.network_mode: ""
  user.user-data: |
    #cloud-config
    package_upgreade: true
    packages:
      - apache2
    timezone: Europe/Stockholm
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  eth1:
    name: eth1
    nictype: macvlan
    parent: eno1
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: proxy

Results

This works fine on my local network. The container obtains an IP address via DHCP on the LAN (eth1) and is also connected to the lxdbr0 network (eth0 - and assigned an IP address there as well). When accessing it from another computer on the LAN it shows the default Ubuntu/Apache web page.

In order to reach the container from the Internet, I added port forwarding in my router, an Ubiquity EdgeRouter X, forwarding port 80 on the “Internet interface” to port 80 on the (eth1) IP address of the container. Still I’m unable to access it from the Internet.

Testing / Trouble Shooting

To make sure port forwarding is at all working as intended, I installed Apache on a local machine on the same LAN and reconfigured port forwarding to point to that IP address instead. Now I could see the default web page when accessing my domain from the Internet. I then created another container with a single network device configured as the macvlan device above. It got its own IP address from the DHCP server on the LAN and after reconfiguring port forwarding again I could reach this container from the Internet.

I then created a copy of the proxy profile above where I changed the order of the network devices so that the macvlan device is listed first as eth0 and lxdbr0 is second as eth1:

...
devices:
  eth0:
    name: eth0
    nictype: macvlan
    parent: eno1
    type: nic
  eth1:
    name: eth1
    nictype: bridged
    parent: lxdbr0
    type: nic
  ...

To my surprise I could now reach this container from the Internet (after reconfiguring port forwarding again of course).

Question

To me this behavior seems strange. Am I missing something or have I stumbled upon something strange. Could this come from limitations/bugs in the (single) NIC on my host (eno1 in the config above) or its kernel modules (I understood that “mileage may vary” based on one of the post linked above)?

Preconditions

  • My host runs Ubuntu 16.04 as do my containers tested here.
  • I don’t use VLANs on my network but I do know my router is VLAN “aware”. I think the switch on my LAN is also capable of VLAN tagging, but this functionality is not enabled.
  • I run LXD 3.0.1 (Xenial backports I think)

(Johannes Midgren) #2

For reference: In my endeavors to make this setup work I earlier posted a somewhat related question in the Ubiquity Community Forums. I only replied to it myself though :blush:


#3

Have you got two default routes in the container, one on each interface? That would explain the problems I think. Configuring lxdbr0 with a static IP address and no gateway/default route in the container should solve the problem I guess.


(Johannes Midgren) #4

Thanks for the reply, @mikma!

Even though I don’t have two default routes in any of these containers, you are on to something. The default route is “associated” with eth0 in both cases, so GW differs:

Original setup (testproxy):

$ lxc exec testproxy route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.191.119.1    0.0.0.0         UG    0      0        0 eth0
10.191.119.0    *               255.255.255.0   U     0      0        0 eth0
172.28.0.0      *               255.255.0.0     U     0      0        0 eth1 

The second test with different order of devices (testproxy2):

$ lxc exec testproxy2 route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.28.0.1      0.0.0.0         UG    0      0        0 eth0
10.191.119.0    *               255.255.255.0   U     0      0        0 eth1
172.28.0.0      *               255.255.0.0     U     0      0        0 eth0

172.28.0.0/16 is my LAN network and 10.191.119.0/24 is the lxdbr0 network.

By simply setting the default route to use my LAN GW in the original setup, it started working!

In testproxy:

# ip route delete default
# ip route add default via 172.28.0.1 dev eth1

Thanks for putting me on the right track! :+1: