Changing ipvlan.mode to l2

yxam · February 3, 2020, 10:31am

here is the setting of the nic (LXD 3.18)

devices:
  eth1:
    nictype: ipvlan
    parent: eth1
    type: nic

ipvlan.mode is l3s by default

# cat /var/snap/lxd/common/lxd/logs/ubu1/lxc.conf | grep ipvlan
lxc.net.1.type = ipvlan
lxc.net.1.ipvlan.mode = l3s
lxc.net.1.ipvlan.isolation = bridge
 
# printf "lxc.net.1.ipvlan.mode = l2" | lxc config set ubu1 raw.lxc -
Error: Invalid config: Only interface-specific ipv4/ipv6 lxc.net. keys are allowed

is there way to change it to l2 mode?

tomp · February 3, 2020, 11:21am

Hi this is not currently possible, although it would be possible to add support for.

Please can I ask what scenario you are operating in that requires l2 ipvlan mode rather than l3s?

Thanks
Tom

yxam · February 3, 2020, 5:13pm

the distributed virtual switch of the vm platform only allows one MAC address coming in/out from the virtual port and i want to have multiple containers running on the host under the same subnet.

i can get around the limitation by setting up linux bridge and some ebtables nat rules to have containers connected within the host. then i learned about ipvlan and think it is a better solution for my use case.

i’ve tried creating ipvlan l2 interface on the host first and passed it to container as phy interface, that worked. but, managing such ipvlan l2 interfaces on the host is still a bit troublesome. if lxd supports ipvlan l2 mode, it will be very neat.

thanks

tomp · February 3, 2020, 5:18pm

OK thanks, and what specifically is the need for l2 mode rather than l3s?

yxam · February 4, 2020, 1:34am

the containers are running on a LXD cluster, it’s not easy to route specific addresses to corresponding cluster members. moreover, there are slight differences on the route table of containers, it is not so scalable to handle routing on the host machine if l3 mode is used.

to have ipvlan l2 mode working makes thing clean, everything looks almost the same as having a linux bridge or OVS on the host machine.

thanks

tomp · February 4, 2020, 2:10pm

Thanks that makes sense, it would allow you to define the IPs inside the container without LXD knowing about them, also if using a DHCP client that supports using client IDs, DHCP could also potentially work too.

You mention that setting up routes is a challenge, however in LXD’s ipvlan l3s mode, we enable proxy ARP/NDP on the specified parent interface (we add a static route to the host’s loopback interface to activate proxy ARP/NDP for the container’s IPs). This means that the wider LAN does not need to route traffic at L3 to the host, and instead the host will respond to ARP/NDP queries for the container’s IPs in order to direct traffic to the host. So you would not necessarily need to configure static routes.

One thing that l3s mode won’t support is DHCP, even when using Client ID, so as long as you don’t need DHCP, then you may be able to still use l3s for your scenario.

yxam · February 5, 2020, 7:03am

thanks a lot. the proxy arp does help to solve the routing issue of the inbound traffic to the containers.

however, mixing the containers traffic(services traffic) with the host machine own traffic(management traffic) is still not optimal. the management traffic needs need to be private while the services traffic probably needs to be quite opened.

i have no idea how difficult it will be to have ipvlan l2 mode working, any pointer where i should look at to see if it’s a task i can manage?

thanks again

tomp · February 5, 2020, 8:47am

It would not be particularly difficult to add l2 support to LXD as the underlying liblxc supports it.

This is the part where the liblxc config is generated:

github.com

lxc/lxd/blob/master/lxd/device/nic_ipvlan.go#L150-L157


	nic := []deviceConfig.RunConfigItem{
		{Key: "name", Value: d.config["name"]},
		{Key: "type", Value: "ipvlan"},
		{Key: "flags", Value: "up"},
		{Key: "ipvlan.mode", Value: "l3s"},
		{Key: "ipvlan.isolation", Value: "bridge"},
		{Key: "l2proxy", Value: "1"},
		{Key: "link", Value: parentName},

This is the l3s part here: https://github.com/lxc/lxd/blob/master/lxd/device/nic_ipvlan.go#L154

If you were using l2 instead, then you would not want this bit: https://github.com/lxc/lxd/blob/master/lxd/device/nic_ipvlan.go#L156

Also we would not need to use proxy arp/ndp, so this function would not need to be called: https://github.com/lxc/lxd/blob/master/lxd/device/nic_ipvlan.go#L189-L215

Also some modification to the config validation would need to be done to introduce a new config key, suggest mode which would default to l3s if not specified, but could be set to either l3s, l3 or l2.

github.com

lxc/lxd/blob/master/lxd/device/nic_ipvlan.go#L62-L110


func (d *nicIPVLAN) validateEnvironment() error {
	if d.inst.Type() == instancetype.Container && d.config["name"] == "" {
		return fmt.Errorf("Requires name property to start")
	}


	if !shared.PathExists(fmt.Sprintf("/sys/class/net/%s", d.config["parent"])) {
		return fmt.Errorf("Parent device '%s' doesn't exist", d.config["parent"])
	}


	extensions := d.state.OS.LXCFeatures
	if !extensions["network_ipvlan"] || !extensions["network_l2proxy"] || !extensions["network_gateway_device_route"] {
		return fmt.Errorf("Requires liblxc has following API extensions: network_ipvlan, network_l2proxy, network_gateway_device_route")
	}


	if d.config["ipv4.address"] != "" {
		// Check necessary sysctls are configured for use with l2proxy parent in IPVLAN l3s mode.
		ipv4FwdPath := fmt.Sprintf("net/ipv4/conf/%s/forwarding", d.config["parent"])
		sysctlVal, err := util.SysctlGet(ipv4FwdPath)
		if err != nil || sysctlVal != "1\n" {
			return fmt.Errorf("Error reading net sysctl %s: %v", ipv4FwdPath, err)

This file has been truncated. show original

Finally the docs would need updating:

github.com

lxc/lxd/blob/master/doc/instances.md#nictype-ipvlan

# Instance configuration
## Properties
The following are direct instance properties and can't be part of a profile:

 - `name`
 - `architecture`

Name is the instance name and can only be changed by renaming the instance.

Valid instance names must:

 - Be between 1 and 63 characters long
 - Be made up exclusively of letters, numbers and dashes from the ASCII table
 - Not start with a digit or a dash
 - Not end with a dash

This requirement is so that the instance name may properly be used in
DNS records, on the filesystem, in various security profiles as well as
the hostname of the instance itself.

This file has been truncated. show original

And an API extension added at the bottom, similar to:

github.com

lxc/lxd/blob/master/doc/api-extensions.md#container_nic_ipvlan

# API extensions

The changes below were introduced to the LXD API after the 1.0 API was finalized.

They are all backward compatible and can be detected by client tools by
looking at the `api_extensions` field in `GET /1.0/`.


## storage\_zfs\_remove\_snapshots
A `storage.zfs_remove_snapshots` daemon configuration key was introduced.

It's a boolean that defaults to false and that when set to true instructs LXD
to remove any needed snapshot when attempting to restore another.

This is needed as ZFS will only let you restore the latest snapshot.

## container\_host\_shutdown\_timeout
A `boot.host_shutdown_timeout` container configuration key was introduced.

It's an integer which indicates how long LXD should wait for the container

This file has been truncated. show original

github.com

lxc/lxd/blob/master/shared/version/api.go#L156


	"kernel_features",
	"id_map_current",
	"event_location",
	"storage_api_remote_volume_snapshots",
	"network_nat_address",
	"container_nic_routes",
	"rbac",
	"cluster_internal_copy",
	"seccomp_notify",
	"lxc_features",
	"container_nic_ipvlan",
	"network_vlan_sriov",
	"storage_cephfs",
	"container_nic_ipfilter",
	"resources_v2",
	"container_exec_user_group_cwd",
	"container_syscall_intercept",
	"container_disk_shift",
	"storage_shifted",
	"resources_infiniband",
	"daemon_storage",

yxam · February 5, 2020, 9:13am

that’s cool. great thanks!