LXD and MACVLAN! HELP ME!

1

Ok, I am starting from scratch. It’s seems the more I google LXD MACVLAN, the more confused I get. I’ve seen at least 3 different ways to configure this and none of them seemed to work for me??

So right now I am sitting at a fresh and updated install of Ubuntu 16.04.04. I have created a partition for ZFS but have not installed it. This is another confusing part because I have seen it as sudo apt-get install zfsutils-linux bridge-utils and without the bridge-utils?? Which one is correct for MACVLAN, if it actually matters?

So, what I need is a simple procedure to configure MACVLAN and one container so the container can access the internet and also be accessed from the internet. Can some supply me with that?

Eventually I need to have 4 containers so hopefully once I have one container up and able to communicate with the internet, the next 3 containers will have no problems.

Thanks and I owe you a beer if I get this running with your help!

2

Fortunately MACVLAN is fairly trivial to setup. We run this with 1k+ containers and it works well.

The way we handle it is we have a profile that we assign to a container with the following info:

devices:
eth0:
name: eth0
nictype: macvlan
parent: hostnet
type: nic
vlan: “123”

This will effectively enable a MACVLAN interface for eth0 inside the container and bind it to the host adapter “hostnet”

Hostnet, in our environment is a bridge, which is tied to a bonded interface of 2 ethernet adapters. You could just have easily created a bridge that corresponds to a single adapter. If you are looking for what the bridge looks like in an interfaces file (ubuntu 16.04), it looks like this:

auto hostnet
iface hostnet inet dhcp
pre-up sleep 6
bridge_ports bond0
bridge_stp off
bridge_fd 0
bridge_maxwait 0

hope this helps, make sure you have support for vlans and bridges in the OS (apt-get install bridge-utils vlan) and running in modules: modprobe 8021q

Hope this helps

1 Like
3

Its useful to note that this just reveals the VLAN to the container, you still have to assign an IP to the interface in the container or let DHCP pick it up. Do NOT set the VLAN tag inside the container, as it will be double tagged and wont work. All other L3 networking stuff can be configured in the container.

And it goes without saying that your switching needs to support it.

1 Like
4

I really appreciate your response, but unfortunately you have my head spinning. Your configuration is nowhere near what mine is. I have one host (VM) and one container. You are talking about multiple ethernet adapters. I need the “trivial” explanation, starting with how to answer all of the lxc init questions?
I still am not sure how to answer the bridging questions?

Maybe you can show me in the following format:

Host network/interface should be this:
Container profile should be this:
Bridge bridge/Container bridge? Do I need a bridge in the host and the container?

5

Here is the trivial case with macvlan, https://blog.simos.info/how-to-make-your-lxd-container-get-ip-addresses-from-your-lan/

You need to note with macvlan:

  1. IF YOU USE A VM (Virtual Machine) FOR THE HOST, THEN IT MIGHT FILTER OUT THE ATTEMPTS OF THE CONTAINERS TO GET AN IP ADDRESS FROM THE LAN. This affects VirtualBox. With VMWare, I think there is a workaround in the settings.
  2. To use macvlan, you do not make any changes to the host or the configuration of LXD. What you do, is create a beautiful lxc profile (assuming a recent LXD version), and attach that profile to the macvlan containers.
  3. For macvlan, you do not install or setup anything that relates to bridge. No bridge at all. Nada.
3 Likes
6

Thanks simos. I will test this out tomorrow.

Ray

7

Why do I find so many different ways to accomplish the same thing?

Here is one example: https://blog.ubuntu.com/2015/11/10/converting-eth0-to-br0-and-getting-all-your-lxc-or-lxd-onto-your-lan

No MACVLAN?

Ray

8

The other thing is running “lxd init”. I think I previously email you about that?

There is a question about bridge. How do I answer that?

Thanks,

Ray

9

So you are telling me that with a fresh install of Ubuntu, all I need is a lxc profile attached to the macvlan containers? What does this profile look like?

And I have no containers as of yet. How do I make them macvlan containers?

10

That post is from 2015, and decribes how to get your containers to obtain an IP from the LAN. They describe one of two ways to do so, using a bridge.
The other way is to use macvlan.

I think I describe well in my post how to use macvlan. My impression is that you go very fast when you try different guides and you end up with some errors.
It helps tremendously if you write in your own words your steps in following a guide.

1 Like
11

@simos

Your blog post worked for me on ubuntu 16.04 with lxc 2.0.11. I can ping google from net1 and can ping net1 from an external computer. I cannot ping net1 from the host, as is expected

Now onto configuring serial devices…

12

@Rayj, can you reply on this? You mention that you use a VM but did not specify which VM. Because macvlan may not work with a VM.

13

Simos,

I am using Ubuntu 16.04.04 on a Vbox VM. LXD version 2.21
VirtualBox Graphical User Interface
Version 5.2.2 r119230 (Qt5.6.2)

Using your procedure, like I said, I am not getting an IP from DHCP.

BUT….here’s the weird thing. If I add eth0 to the container interfaces as auto eth0, iface eth0 inet dhcp, I discovered that if I then run tcpdump from the VM host on interface (enp0s3), the container magically gets an IP from DHCP. However I cannot ping anything from inside the container? And I cannot ping the container from outside.

It doesn’t make sense to me why running tcpdump on the VM causes the container to receive an IP? The host VM IP is 192.168.0.40 and the container IP gets 192.168.0.41!
However, I can’t ping the container from my PC nor from the VM. I get “Destination Unreachable”.
I can’t ping the gateway, 192.168.0.1, from the container either.

Here is the container ifconfig after running tcpdump on the VM host:

root@LPC1:~# ifconfig
eth0 Link encap:Ethernet HWaddr 00:16:3e:32:ba:fd
inet addr:192.168.0.41 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::216:3eff:fe32:bafd/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:313 errors:0 dropped:0 overruns:0 frame:0
TX packets:35 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:35935 (35.9 KB) TX bytes:3514 (3.5 KB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:14 errors:0 dropped:0 overruns:0 frame:0
TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:1236 (1.2 KB) TX bytes:1236 (1.2 KB

Here is the container ip route after running tcpdump on the host VM:

root@LPC1:~# ip route
default via 192.168.0.1 dev eth0
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.41

Here is the container ifconfig after restarting the container:

root@LPC1:~# ifconfig
eth0 Link encap:Ethernet HWaddr 00:16:3e:32:ba:fd
inet6 addr: fe80::216:3eff:fe32:bafd/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:25 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:2310 (2.3 KB) TX bytes:1534 (1.5 KB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

Here is the container ip route after restarting the container:

root@LPC1:~# ip route
root@LPC1:~#

So, restarting the container, the IP goes away and “ip route” shows nothing.
Run tcpdump on the VM and the IP comes back?

It seems that this is a Vbox issue, but is there an easy way to see if this really is a Vbox issue?

Any ideas? Work arounds?

Thanks,

Ray

14

My post on setting up macvlan with LXD does not apply when you use virtualization like VMWare or VirtualBox. In addition, you must use a LAN (not WiFi) on the host for this to work.

As far as I understand, if you use either VMWare or Virtualbox, you need to tinker with the Networking Advanced Settings (Setting relating Promiscuous mode) so that macvlan with LXD works. I have not used these and from what I read, someonehttps://www.opentechguides.com/images/howto/howto_14001.png managed to get it to work with VMWare.
I am glad you are trying with VirtualBox and I hope you can make it work.

The setting that you need to try out, is shown in the following screenshot (source: opentechguides.com). It’s the greyed out Promiscuous mode. I do not know why it is greyed out in that screenshot. If it is greyed out for you as well, then you need to fix that first.

15

I am using a LAN and not WiFi. And I set Promiscous mode to “Allow All”.
Still have the same issue.

What did you think about the tcpdump weirdness with the container?

Ray

16

macvlan with VirtualBox should just work, when you have Promiscuous Mode enabled.

When you are running sudo tcpdump on an interface, the interface enters PROMISC mode for the duration of you running tcpdump.
You can also change the mode of the interface with

sudo ifconfig enp0s3 promisc

and it will stay that way as long as the interface is up.

Others have reported that they managed to get it to work with VMWare, but when they tried with VirtualBox, they could not make it work.
That leads me to believe that it’s a bug in VirtualBox.
What can you do now? Have a look at virtualization - Assign multiple IP addresses in NAT mode to the same VM - Server Fault A person there asks the correct question (but does not receive a relevant answer). Work on that question and ask at VirtualBox on Windows Hosts - virtualbox.org

1 Like
17

I am following your procedure for MACVLAN. My Default profile looks nothing like yours?

Here is my default profile:

lxc profile show lanprofile

config:

environment.http_proxy: “”

user.network_mode: “”

description: Default LXD profile

devices: {}

name: lanprofile

used_by: []

Do I need to run LXD INIT to get the default profile created first? If so, how do I answer the bridge questions?

And your device names seem to be incorrect in your procedure?  Your ip route shows:  $ **ip route show default 0.0.0.0/0**

default via 192.168.1.1 dev ***enp5s12*** proto static metric 100

But in your lanprofile you show parent: enp3s0?

Or am I wrong?

Ray

18

I have made some typos.

First, indeed at my post at https://blog.simos.info/how-to-make-your-lxd-container-get-ip-addresses-from-your-lan/ I mention a weird interface enp3s0 while the rest of the post was talking about enp5s12. I tried the tutorial on two systems to make sure it works, and that occurrence of enp3s0 was a remnant from the other system. I have now updated my post and fixed this error.

Second, I mention above that macvlan should work with VirtualBox. Indeed, VirtualBox should work with LXD+macvlan when you enable the Promiscuous mode, but it has not been reported yet to work!
I have not spend time to test VirtualBox with LXD+macvlan, therefore, someone else needs to test and report back. However, others have mentioned that LXD+macvlan works on a VMWare Linux VM.

In your reply immediately above, you show a lanprofile that is empty and has no configuration.
Most likely you created it with lxc profile create lanprofile. However, in my blog post, I used a trick. Specifically, I copied the default profile onto the lanprofile. You do now show the default profile, therefore I do not know whether your default profile is problematic. But if your default profile has no network configuration at all, then no container (even non-macvlan ones) would be able to get any network configuration.

19

Super helpful, well written blog post, Simos!