Multiple LXD containers on single macvlan interface

LXD
1

I’m a little confused as to how the following scenario works. It’s a very simple setup, so I hope the explanation is simple.

I have a host with a single physical NIC. I create a single macvlan sub-interface in bridge mode off this physical NIC. Then I start up two LXD/LXC containers. Each with their own unique MAC and IP. In the profile, I specify the same single macvlan sub-interface as each container’s parent interface.

Both containers have access to the network without issue. I’m also able to SSH into each container using each container’s unique IP address (from a different machine, not the machine hosting the containers). This is the bit that confuses me:

How is all of this working underneath the hood? How are two containers using a single macvlan interface? Isn’t there going to be some sort of collision with MAC/IP? Shouldn’t this not work? Shouldn’t I need one macvlan subinterface per container?

macvlan isn’t documented much, hoping someone out there can help out.

Second side question:
Am I right to assume that a macvlan interface uses virtual functions on a NIC?

2

The exact way the kernel level macvlan code will behave is pretty NIC dependent.
In most cases, all it does is update the MAC table in the NIC to listen for additional MACs, then forward frames headed to those MACs directly into the virtual macvlan interface.

That effectively saves you from having to use PROMISCUOUS mode on the NIC, so long as you don’t exceed the MAC table size.

Because of where it hooks into the kernel though, macvlan devices cannot talk to anything that’s bound on the parent device. This tends to be the most annoying limitation of it.

As for what LXC/LXD is doing in your case, I’m a bit confused as to your exact setup.
If you’re using LXD, can you post:

  • lxc network list
  • lxc config show --expanded NAME (for your two containers)

That should help me understand your actual setup.

1 Like
3

Hello,

Thanks for the information. Sounds like the NIC will fwd frames to either container. Which would tell me this is working as it should.

I don’t see the “lxc network list” command. I’m on vesion 2.0.10

Here is “config show” has on the two containers:
macvlan is the name of the macvlan interface on the host. It appears at macvlan@eth2 in the host. Both containers use this interface for connectivity.

ubuntu:~$ sudo lxc config show --expanded u1
architecture: x86_64
config:
  raw.lxc: lxc.aa_profile=unconfined
  security.nesting: "true"
  user.network_mode: link-local
  volatile.base_image: 46d9dc87a1f72ca828294cf212e4ee8d4f9c1fa2fedd0fbf42ee57e1407d15e4
  volatile.eth0.hwaddr: 52:54:aa:c0:5d:01
  volatile.idmap.base: "0"
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: RUNNING
devices:
  eth0:
    name: eth0
    nictype: macvlan
    parent: macvlan
    type: nic
  root:
    path: /
    type: disk
ephemeral: true
profiles:
- default
stateful: false
ubuntu:~$ sudo lxc config show --expanded u2
architecture: x86_64
config:
  raw.lxc: lxc.aa_profile=unconfined
  security.nesting: "true"
  user.network_mode: link-local
  volatile.base_image: 46d9dc87a1f72ca828294cf212e4ee8d4f9c1fa2fedd0fbf42ee57e1407d15e4
  volatile.eth0.hwaddr: 52:54:aa:c0:5d:03
  volatile.idmap.base: "0"
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: RUNNING
devices:
  eth0:
    name: eth0
    nictype: macvlan
    parent: macvlan
    type: nic
  root:
    path: /
    type: disk
ephemeral: true
profiles:
- default
stateful: false
4

Ok, so you’re doing two levels in macvlan with the config above.

You could go with something simpler with just:

  eth0:
    name: eth0
    nictype: macvlan
    parent: eth0
    type: nic

That way you don’t have to manually create the “macvlan” device on the host.

With your current config, you’re actually using two levels of macvlan. The first one is the one you created by hand, then LXD sets up a second macvlan layer on top of it. It works, but it’s not quite as efficient :slight_smile:

5

I see. Just to be clear here. With my current setup, there are basically 3 macvlan interfaces. the one I created, plus one for each container which uses the one I created as a parent? Therefore, I can get it down to just 2 if I use your setup.

correct?

6

Yep, that’s correct.

7

Thanks a lot!

This now makes sense with other stuff I’ve read. My universe is whole again.