Weekly status #278

News
, , , ,
1

Weekly Update Templates
Weekly Update Templates960Ă—540 53.5 KB

Weekly status for the week of the 5th December to the 11th December.

Introduction

The highlight of the past week was the release of LXD 5.9. As its the last release of the year we primarily focused on bug fixes and performance improvements. For more details please see the release notes and release video:

https://www.youtube.com/watch?v=6aFFVj3W5oI

MicroCloud demo

Hot on the heels of the MicroCeph demo last week, this week @stgraber has made a video demonstrating the new MicroCloud package we have been working on.

The MicroCloud snap drives two other snaps (LXD and MicroCeph) enabling automated deployment of a highly available LXD cluster for compute with Ceph as a storage backend.

For networking, MicroCloud will use a default network bridge, but MicroOVN is in development and will be added once completed.

https://www.youtube.com/watch?v=ZSZoLnp-Ip0

For more info see Introducing MicroCloud

Job openings

Canonical Ltd. strengthens its investment into LXD and is looking at building multiple squads under the technical leadership of @stgraber.

As such, we are looking for first line managers (highly technical) and individual contributors to grow the team and pursue our efforts around scalability and clustering.

All positions are 100% remote with some travel for internal events and conferences.

For more info please see LXD related openings at Canonical Ltd (2022-2023)

LXD

Improvements:

  • Restructuring and cleaning up the instance devices documentation.
  • With the work done to support QEMU 6.0 in LXD 4.15 we started using QMP to effectively hot-plug disks just before the VM guest was started. This was done because using QMP is preferred by upstream rather than using a config file. It was also required in order to support hot-plugging of disks. However due to an issue in QEMU this caused the boot priority of disks to not be respected (because the UEFI firmware had already initialised). In order to workaround this issue we had to reset the VM after the initial disks were added by sending a system_reset. Unfortunately this introduced another issue, which was that this wasn’t possible when using -no-reboot flag that LXD was using to ensure that if a VM guest self-rebooted that the normal stop and start process was run by LXD (to be consistent with doing lxc restart <instance>). At the time the solution was to remove the -no-reboot flag and have LXD catch the RESET event from a VM that was self-rebooting and initiate a forced stop and then start. This was not ideal as LXD was effectively racing the VM guest’s own reboot. We have now changed the approach so that LXD will now use the set-action QMP command to instruct QEMU to shutdown when a VM guest self-reboots. This effectively restores the behaviour of the -no-reboot flag, but allows us to set it after the system_reset that is needed to respect the boot priority of devices.

Bug fixes:

  • Introduced limits on the concurrency of the metrics API endpoint when it is generating metrics. Previously it spawned one Go routine per running instance, and thus had unbounded potential concurrency which could then incur a lot of load on the host system and impact workload performance. Now the concurrency is limited to the number of CPUs. There were also various other efficiency improvements made to the metrics API endpoint.
  • Fixed regression in the operation of vsock (used by the lxd-agent) with nested VMs.
  • Fixed an issue where errors coming from ethtool in the resources API were preventing the endpoint from working. We now ignore errors from that data source and make the data it provides optional, as it was failing in a number of different scenarios and wasn’t reliable.
  • Fixed an issue with Ceph to Ceph optimized migrations leaving behind the temporary snapshots on the source system. The fix was to add a check that disables the multi-sync mode when optimized migration has been negotiated (as there is no need as optimized migration uses snapshots).
  • Fixed two regressions related to macvlan NICs on VMs.
    • Firstly the introduction of multi-queue macvlan NICs in LXD 5.6 caused an issue where QEMU would deactivate the NICs when LXD disconnected from the QEMU monitor (such as when it was reloaded). This was investigated and found to be due to the way that the macvtap file descriptors were passed to QEMU as an fdset using the add-fd command. It seems that unless the passed file descriptors are referenced in the VM config using /dev/fdset/{id} that QEMU will clean up the fdset when LXD disconnects. To workaround this issue we have switched to using the getfd QMP command that allows for passing file descriptors without using fdsets.
    • Secondly, the introduction of VM CPU hotplugging in LXD 5.8 prevented the multi-queue NIC support for VMs from working. This is because previously this was based on the number of CPUs present at QEMU process start. When CPU hotplugging got introduced the initial CPU count is now always 1 and the additional CPUs are hotplugged shortly before the VM guest is started. This meant that NICs were always using 1 queue. This has been fixed so that NICs now query QEMU via QMP to ascertain the number of CPUs at the time the NIC is added, be that at start time or later if it is hotplugged.
  • Fixed an issue where VMs that self-rebooted would sometimes fail to restart due to In Use unmount storage errors. These specific errors are now ignored as it is valid for another operation to be using the instance’s volume (such as the metadata API endpoint) at the time the VM is rebooting and so should not prevent it.

LXCFS

Bug fixes:

  • Respect --prefix flag when installing systemd unit files.
  • Fix possible use-after-free in find_proc_stat_node.

Distrobuilder

Improvements:

  • Allow specifying compression level.
  • Increase TLS handshake timeout for downloaded.

YouTube videos

The LXD team is running a YouTube channel with live streams covering LXD releases and weekly videos on different aspects of LXD. You may want to give it a watch and/or subscribe for more content in the coming weeks.

https://www.youtube.com/lxd-videos

Contribute to LXD

Ever wanted to contribute to LXD but not sure where to start?
We’ve recently gone through some effort to properly tag issues suitable for new contributors on Github: Easy issues for new contributors

Upcoming events

Ongoing projects

The list below is feature or refactoring work which will span several weeks/months and can’t be tied directly to a single Github issue or pull request.

  • Stable release work for LXC, LXCFS and LXD

Upstream changes

The items listed below are highlights of the work which happened upstream over the past week and which will be included in the next release.

LXD

LXC

LXCFS

Distrobuilder

LXD Charm

  • Nothing to report this week

LXD Charm

  • Nothing to report this week

Distribution work

This section is used to track the work done in downstream Linux distributions to ship the latest LXC, LXD and LXCFS as well as work to get various software to work properly inside containers.

Ubuntu

  • Nothing to report this week

Snap

  • lxd: Bump to 5.9
  • lxd: Cherry-pick upstream bugfixes
  • lxcfs: Cherry-pick upstream bugfixes
1 Like