Weekly status for the week of the 9th of November to the 15th of November.
Introduction
The vast majority of the past week was focused on fixing storage bugs and improving our cgroupv2 support in LXD.
LXD
The storage changes can be broadly broken down into these areas:
- Added per-volume mount/unmount locks to prevent multiple operations trying to perform a mount operation concurrently.
- Improved volume in-use detection to prevent changes to volumes when they are attached to running instances.
- Added reference counting for each ongoing operation that requires a volume to be mounted so that when an operation finishes and requests the volume to be unmounted this can be prevented if there is another ongoing operation still using the volume.
- Re-ordered container device cleanup on container stop so that container is fully stopped before volume are unmounted.
- Modify container stop hooks to use container and project name rather than container ID in order to allow clean container shutdown after being recovered using
lxd importwhile still running (which causes the container ID to be changed).
Related to that last point we have now introduced the volatile.uuid config key for all instance types (containers and VMs). This replaces the volatile.vm.uuid key for VMs (a patch will copy the existing value into the new key so it doesn’t change). It is generated automatically on first instance start and persisted for the lifetime of the instance. It can be used as a persistent reference to the instance that will survive recovery via lxd import that causes the instance’s ID to change.
The mount referencing counting feature was added primarily to allow VM snapshots on ZFS to be used as the source for copying to a new instance. This was not working before because ZFS requires the parent volume to be activated before its snapshots can have their block devices activated. This meant that in order to mount the snapshot to copy it we need to also mount the parent. Unfortunately the existing storage layer could not account for tracking ongoing operations across multiple volumes, and attempting to unmount the parent volume when unmounting the snapshot would have caused problems. However this new feature also fixes a class of problems related to long running file transfer operations when stopping/starting a container during a file transfer operation.
We have also added rules to prevent the use of quotes in project names as this was causing issues with some storage engines due to the fact that the project name is used in the volume name and some storage drivers don’t allow quotes in the volume name.
As part of improving our cgroup abstraction and automated testing work, several bugs in that area were fixed as well.
The lxc command also gained two new features. Firstly, two new columns have been added to the lxc list command; these are M Memory usage (%) and u CPU usage (in seconds).
Secondly, the lxc snapshot command gained a -reuse flag allowing for an existing snapshot to be deleted and re-created using a single command.
Distrobuilder
A new copy generator was added to distrobuilder to allow files from the host to be copied into the container. See https://github.com/lxc/distrobuilder/blob/master/doc/generators.md#copy for more information.
Youtube channel
We’ve started a Youtube channel with a couple of live streams so far.
You may want to give it a watch and/or subscribe for more content in the coming weeks.
Contribute to LXD
Ever wanted to contribute to LXD but not sure where to start?
We’ve recently gone through some effort to properly tag issues suitable for new contributors on Github: Easy issues for new contributors
Upcoming events
- Nothing to report this week
Ongoing projects
The list below is feature or refactoring work which will span several weeks/months and can’t be tied directly to a single Github issue or pull request.
- Distrobuilder Windows support
- Virtual networks in LXD
- Various kernel work
- Stable release work for LXC, LXCFS and LXD
Upstream changes
The items listed below are highlights of the work which happened upstream over the past week and which will be included in the next release.
LXD
- Storage: Fix custom volume attach and resource usage reporting when volumes exist on multiple nodes
- Fix blkio.weight cgroup handling
- Storage: Further bug fixes in relation to volume used by detection
- Storage: Ensure Location field is empty when storage driver is remote in storagePoolVolumeGetType
- Storage: Always initialise db.StorageRemoteDriverNames in DB tests
- Storage: Adds volume mount & unmount locking
- Container: Stop non-NIC devices after container fully stopped
- doc/rest-api: auth property is never set to guest
- lxd/apparmor: Workaround socket handling
- lxd/storage: Expand local config
- lxd/cgroup: Fix swap limits
- lxc: Add ability to reuse exisitng snapshot name
- Storage: Remove instance configure internal function
- Final bits of CGroupV2 support
- Storage: Mount reference counting
- lxd-agent: Don’t allow connections when rebooting
- lxc list help message typo fix
- lxc list, two new usage columns
- Project: Reject quotes in project names
- Container: Pass name rather than ID to LXC start, stopns and stop hooks
- lxd/storage: Apply rename template
- Instance: Adds volatile.uuid key for container and VMs, replaces volatile.vm.uuid for VMs
- doc: fix typos in instances.md
- lxd/instance/lxc: Add extra check for devpts_fd
LXC
LXCFS
- Nothing to report this week
Distrobuilder
- generators: Add copy generator
- generators: update doc
- Added missing definitions in generators hosts, hostname
Distribution work
This section is used to track the work done in downstream Linux distributions to ship the latest LXC, LXD and LXCFS as well as work to get various software to work properly inside containers.
Ubuntu
- Nothing to report this week
Snap
- criu: Bump to 3.15
- nft: Bump to 0.9.7
- libnftnl: Bump to 1.1.8
- squashfs: Bump to 1.0.3
- libtpms: BUmp to 0.7.4
- ovn: Bump to 20.09.0
- spice-protocol: Bump to 0.14.3
- lxcfs: Cherry-picked upstream bugfixes
- lxd: Bump to 4.8
- lxd: Cherry-picked upstream bugfixes