We are running a bunch of Ubuntu 16 servers with LXD 2.14 (and beyond) to host a bunch of containers. Each LXD server is running as a VMWare ESX virtual machine, and we are configuring two NICs down to the server. The first NIC is the management interface, and the second NIC is a TRUNK port passing two VLANs via “mac-vlan” config. Everything is working great with this config.
We have recently isolated a network performance issues with these TRUNK ports because they are set to promiscuous mode. Disabling promiscuous on the port solves the performance issue but the containers no longer communicate with the network.
The container profile looks like this:
$ lxc profile show Dual_Network
config: {}
description: ""
devices:
eth0:
name: eth0
nictype: macvlan
parent: eth1.200
type: nic
eth1:
name: eth1
nictype: macvlan
parent: eth1.201
type: nic
eth2:
name: eth2
nictype: macvlan
parent: eth1.202
type: nic
name: Dual_Network
used_by:
Is there another network option we can use to provide VLAN trunking to the LXD servers without requiring mac-vlan?
I think that’s more of a VMWare question than a LXD question.
PROMISCUOUS mode usually means that all switch traffic is replicated to all member, effectively turning the switch into a hub. That’s obviously terrible for performance and likely explains what you noticed.
LXD containers don’t need the parent device to be in promiscuous mode, they’re not an IDS or similar software that does need a copy of all traffic.
But the reason why you needed promiscuous mode to be enabled is because turning this on, also allows the virtual machine to advertise multiple MACs on the network interface, which is necessary for containers to work (as each will have its own MAC address).
Now hopefully there is some setting in VMWare that lets you disable port security, allowing for multiple MACs on the same network interface without also requiring you to turn on full promiscuous mode.
If you can find that, then that’ll fix your problem. If that’s not possible, then you’re pretty much left with 3 options:
Promiscuous mode and its performance impact
Pass one network interface per container and use the physical nictype in LXD
Don’t use L2 networking, route a subnet to each of your hosts and have each host provide DHCP for that subnet in LXD. Depending on the design of your network, this may be very difficult to do though.
It seems this issue has been around for a long time - more specifically affecting the “nested” install for VMWare. Based on what I have read, there is a “fling” (plug-in) that can be installed on the ESX servers to address this problem. It can be found here
As to your points above, option #2 from your list won’t work due to the max number of physical ports that can be connected to a VM. And, as you mentioned, option #3 will be difficult to implement in our environment since we have been in production for a while. It seems we are stuck with promiscuous mode and the performance impacts. I will try to install the VMWare “fling” and see if that helps.