Macvlan without GW

LXD
1

lxd v. 4.15

Multiple hosts, each connected via eno1 / static IP to public internet.
Each host has an additional 10G interface ens1f0, without default gateway.
Physically connected via Cables and 10G switch, isolated, no public uplink.

Host3:

ens1f0:
  dhcp4: no
  addresses: [ 192.168.1.3/24]
  # No Gateway

Host4:

ens1f0:
  dhcp4: no
  addresses: [ 192.168.1.4/24]
  # No Gateway

Interconnection works so far, as isolated network.

Now I am trying to add interfaces eth1 in containers of each host in order to extend this isolated network into selected containers.

lxc profile add Macv
lxc profile device add Macv eth1 nic nictype=macvlan parent=ens1f0

Host3 Container31:

eth0: bridged network for ingress/egress over public IP
eth1:

iface eth1 inet static
address 192.168.1.31
netmask 255.255.255.0
#NO GW

Host4 Container41:

eth0: bridged network for ingress/egress over public IP
eth1:

iface eth1 inet static
address 192.168.1.41
netmask 255.255.255.0
#NO GW

I can from container 41(192.168.1.41) ping host3(192.168.1.3) and the container31(192.168.1.31) in it.
from container31(192.168.1.31) ping host4(192.168.1.4) and the container41(192.168.1.41) in it.
But cant ping from containers their own hosts:
container31(192.168.1.31) cant ping host3(192.168.1.3), container41(192.168.1.41) cant ping host4(192.168.1.4).

Do I need to create a fake GW, as route for all of them?
But then, by adding GW to container interface, it creates new routes and by default lose the uplink connection, as it uses that GW with same weight for in/outbound traffic everywhere.

Or rather use bridge for above scenario?

2

Appreciate any feed back on this

3

macvlan instances can’t communicate with their host. See Linux Containers - LXD - Has been moved to Canonical

macvlan effectively lets you fork your physical NIC, getting a second interface that’s then used by the instance. This saves you from creating a bridge device and veth pairs and usually offers better performance than a bridge.

The downside to this is that macvlan devices while able to communicate between themselves and to the outside, aren’t able to talk to their parent device. This means that you can’t use macvlan if you ever need your instances to talk to the host itself.

In such case, a bridge is preferable. A bridge will also let you use mac filtering and I/O limits which cannot be applied to a macvlan device.

4

@ matjaz
Thanks for elaborating.
I must have missed the additional note in docs, where this behavior was described. Also other instructions in web are not mentioning the downside or it is suppressed by all enthusiasm about Macvlan.
It makes it in my case useless, also ipvlan seems to behave similarly. But it might be a handy feature for some other use case, where direct communication between host and guest are intentionally not wished.

Yes, indeed bridge might be the solution in my case, yet the host is highly bridged everywhere and I wanted a simpler solution for an internal connection without public uplink or eventually dhcp server etc.

5

There is a workaround:

Create manually another Macvlan on same iface on host:

ip link add macv link ens1f0 type macvlan mode bridge
ip addr add 192.168.1.1/32 dev macv
ip link set macv up
ip route add 192.168.1.0/24 dev macv

Of course can go for smaller subnets /27 or /28 and GW and container IP within that range, when you dont need a /24 network for that purpose.
Make it persistent if wished (OS/Network Specific routine)

Than use Host’s MacVlans address 192.168.1.1 as GW for Container’s Macvlan interface…