Weekly status #178

tomp · December 14, 2020, 5:42pm

Weekly status for the week of the 7th of December to the 13th of December.

Introduction

The highlight of the past week was the release of LXD 4.9.

Please see the release notes for more information.

Over this past week we have added new information to the Resources and LXD APIs, allowed user custom config keys on all configuration objects, added instance limits to projects and have added the concept of a routed ingress mode for OVN networks. As well as the usual round of improvements and fixes, including a VM specific continuous integration suite targeting storage related activities.

LXD

API Additions

The Resources API has been extended to include the following new information:

Adds the usb_address field to the network card entries.
Adds the usb_address and pci_address fields to the disk entries.

The LXD API has also been extended by adding the a vlan section to the /1.0/networks/NAME/state API.

You can now specify custom config keys prefixed with user. to everywhere that config can be stored.

Project Instances Limits

There is a new project config key limits.instances. This key will count containers and virtual-machines and so act as an overall limit on top of the existing container/virtual-machines limits.

OVN Networks Routed Ingress Mode with Anycast

Previously OVN private networks could be connected to an “uplink” network (either a LXD managed bridge or physical port). Outbound traffic would then be NATted to OVN gateway router’s IP on the uplink network. You could then optionally define a set of “external subnets” on the uplink which then could then be selectively used by the OVN networks and associated instance NICs in order to allow an OVN network to be reachable inbound from the external network (either by directly assigning all or part of an external subnet as the OVN network’s subnet, or by routing all or part of the subnet to an instance NIC device).

In order to “publish” the assigned subnet IPs onto the uplink network we setup the OVN virtual gateway router to respond to ARP and NDP solicitations on the uplink network, allowing traffic for the external IPs to flow into the OVN network. In this way the OVN router is acting as a layer 2 proxy.

Although this is a useful approach for many networks and doesn’t require any special support by the uplink router, it has its restrictions. Because each IP in the external subnet has to be published individually using a DNAT rule, in order to avoid creating too many rules we instigated a limit on the size of a subnet that can be routed to a specific instance NIC (/26 for IPv4 and /122 for IPv6).

We have now added the concept of a “routed” ingress mode to the physical network type, which then alters how OVN networks that use that network as an uplink publish their external IPs.

The physical network config key is ovn.ingress_mode and can be set to either l2proxy (the default) or routed.

When using an uplink in “routed” ingress mode OVN networks no longer publish their external subnet IPs using DNAT and so are not resolvable using ARP/NDP solicitations. Instead we expect a separate process to use the LXD API to publish the external subnets using a routing protocol to one or more upstream routers. When in this mode, we also remove the size restrictions for instance NIC’s external subnets. In addition to that we have also introduced the physical uplink boolean config keys ipv4.routes.anycast and ipv6.routes.anycast that when set to “true” will also remove the subnet overlap checks that prevent using the same subnet (or part of it) in multiple networks or instance NICs. This allows for the use of anycasting the same subnet into two different OVN networks.

We have also produced a proof-of-concept standalone daemon called lxd-bgp that will use the LXD API and then publish the OVN external subnets to an upstream router using BGP.

See https://snapcraft.io/lxd-bgp and https://github.com/stgraber/lxd-bgp

We have also added support for ipv4.dhcp and ipv6.dhcp boolean options on OVN networks, allowing DHCP to be disabled if needed.

Storage

A new continuous integration test suite has been added (https://github.com/lxc/lxc-ci/blob/master/bin/test-lxd-vm) to explicitly check VM storage related operations (such as specifying volume size both from the pool and from the profile, checking resize is applied correctly, and that copying between pools maintains volume size).

In doing so several bugs have been identified and fixed mainly related to differences in the way that different storage pool drivers round the specified volume size.

An issue added in LXD 4.8 has been fixed that caused a 10s when mounting a VM when using ZFS and Snap. This could cause delays in lxc ls as well as slowing down VM start up.

Cluster storage pool and network setup

We have also been working on improving the way that storage pools and networks are setup in clusters. When adding a storage pool or network when using an LXD cluster it is required that each cluster member be configured with the new item first (using the --target argument) and then the global configuration is applied after that. This introduces the possibility that the pool or network can be successfully added to one member but fail on another due to a local situation on a specific member.

Previously if any members failed then the pool or network was marked as “Errored” and could not be modified. It was also possible that if LXD was restarted with an Errored storage pool that it would not start up on the affected members.

We have now modified the creation process to track which individual members have succeeded in creating the item and which have failed. We then allow the failed members to have their config updated and the global creation re-run (we skip members that have already been setup).

This new process also allows errored storage pools and networks to be deleted (and then re-created) if the problem cannot be resolved with local config changes, and we do not block LXD start up with errored pools or networks.

LXC

A fix has been added to restore network interface to the correct namespace on error. There have also been several tree-wide code cleanups and bug fixes.

LXCFS

Support for emulating /proc/slabinfo with slab accounting memcg as been added.

Youtube channel

We’ve started a Youtube channel with a couple of live streams so far.

You may want to give it a watch and/or subscribe for more content in the coming weeks.

Contribute to LXD

Ever wanted to contribute to LXD but not sure where to start?
We’ve recently gone through some effort to properly tag issues suitable for new contributors on Github: Easy issues for new contributors

Upcoming events

Nothing to report this week

Ongoing projects

The list below is feature or refactoring work which will span several weeks/months and can’t be tied directly to a single Github issue or pull request.

Distrobuilder Windows support
Virtual networks in LXD
Various kernel work
Stable release work for LXC, LXCFS and LXD

Upstream changes

The items listed below are highlights of the work which happened upstream over the past week and which will be included in the next release.

LXD

LXC

LXCFS

proc_fuse: add /proc/slabinfo with slab accounting memcg

Distrobuilder

Nothing to report this week

Distribution work

This section is used to track the work done in downstream Linux distributions to ship the latest LXC, LXD and LXCFS as well as work to get various software to work properly inside containers.

Ubuntu

Nothing to report this week

Snap

lxd: Bumped to 4.9
libusb: Bumped to 1.0.24
sqlite: Bumped to 3.34.0
qemu: Bumped to 5.2.0
edk2: Fixed some submodule issues
qemu: Fixed build under meson
lxd-migrate: Fixed migrating network count
lxd: Cherry-picked upstream bugfixes