My issue is that I’m only able to reach a container with two network devices from Internet using port forwarding on the router if I use a specific order of the defined network devices. The macvlan device has to come first.
Background
I would like to set up a number of web services using LXD. My idea is to use one container as an HTTP reverse proxy (defining several vhosts in apache and handle all SSL) and then install various services such as NextCloud, MythWeb and ZoneMinder in separate containers. Communication between the containers could use the default lxdbr0 and the reverse proxy should communicate with the outside world using macvlan.
After reading a few topics here, and especially the following two great replies by @stgraber, I thought I understood a bit more about how a macvlan setup works:
Configuration
I use the following profile to create the basics of the reverse proxy container:
config:
environment.http_proxy: ""
user.network-config: |
version: 1
config:
- type: physical
name: eth0
subnets:
- type: dhcp
ipv4: true
control: auto
- type: physical
name: eth1
subnets:
- type: dhcp
ipv4: true
control: auto
user.network_mode: ""
user.user-data: |
#cloud-config
package_upgreade: true
packages:
- apache2
timezone: Europe/Stockholm
description: Default LXD profile
devices:
eth0:
name: eth0
nictype: bridged
parent: lxdbr0
type: nic
eth1:
name: eth1
nictype: macvlan
parent: eno1
type: nic
root:
path: /
pool: default
type: disk
name: proxy
Results
This works fine on my local network. The container obtains an IP address via DHCP on the LAN (eth1) and is also connected to the lxdbr0 network (eth0 - and assigned an IP address there as well). When accessing it from another computer on the LAN it shows the default Ubuntu/Apache web page.
In order to reach the container from the Internet, I added port forwarding in my router, an Ubiquity EdgeRouter X, forwarding port 80 on the “Internet interface” to port 80 on the (eth1) IP address of the container. Still I’m unable to access it from the Internet.
Testing / Trouble Shooting
To make sure port forwarding is at all working as intended, I installed Apache on a local machine on the same LAN and reconfigured port forwarding to point to that IP address instead. Now I could see the default web page when accessing my domain from the Internet. I then created another container with a single network device configured as the macvlan device above. It got its own IP address from the DHCP server on the LAN and after reconfiguring port forwarding again I could reach this container from the Internet.
I then created a copy of the proxy profile above where I changed the order of the network devices so that the macvlan device is listed first as eth0 and lxdbr0 is second as eth1:
...
devices:
eth0:
name: eth0
nictype: macvlan
parent: eno1
type: nic
eth1:
name: eth1
nictype: bridged
parent: lxdbr0
type: nic
...
To my surprise I could now reach this container from the Internet (after reconfiguring port forwarding again of course).
Question
To me this behavior seems strange. Am I missing something or have I stumbled upon something strange. Could this come from limitations/bugs in the (single) NIC on my host (eno1 in the config above) or its kernel modules (I understood that “mileage may vary” based on one of the post linked above)?
Preconditions
- My host runs Ubuntu 16.04 as do my containers tested here.
- I don’t use VLANs on my network but I do know my router is VLAN “aware”. I think the switch on my LAN is also capable of VLAN tagging, but this functionality is not enabled.
- I run LXD 3.0.1 (Xenial backports I think)
