Weekly status for the week of the 24th of May to the 30th of May.
Introduction
This past week has seen the addition of three new RHEL derived distributions in distrobuilder, several improvements and bug fixes in LXD, performance improvements in dqlite snapshotting and nftables firewall support in lxc-net.
Moving our IRC presence to Libera Chat
Our IRC (live chat) channels have moved from Freenode over to Libera Chat.
You can now find us in #lxc and #lxc-dev on irc.libera.chat.
A web chat client can be found here: Kiwi IRC
The LXD team is hiring
The LXD team at Canonical is currently looking for a Go software engineer to join our distributed team of engineers. We’re looking for candidates anywhere in Europe or the Americas!
All LXD positions are 100% remote with some travel for internal events and conferences.
LXD
The lxc monitor --pretty command now supports lifecycle events (as it previously only supported --logging type events). In addition to that, we now also display the requester as part of the lifecycle event.
The persistent warnings feature has had two new warning types added to it; AppArmor disabled and cluster time skew.
An issue that was preventing VMs being started when using additional disks attached with readonly=true has been fixed. The issue was that the LXD applied AppArmor profile was preventing QEMU from accessing the disk share directly. For writeable disks shared to the VM guest using 9p we use the virtfs-proxy-helper process to allow QEMU to access host directories outside of the main AppArmor profile. However originally when adding VM disk device support we found a bug in QEMU that meant when using virtfs-proxy-helper that the readonly property was not respected, so instead we got QEMU to access these directories directly for these types of disk. Subsequently the AppArmor profile was introduced which broken this behaviour. However that QEMU issue has since been fixed and we can now use virtfs-proxy-helper for both writeable and readonly disks. During the process of fixing this it was also found that when using the faster virtio-fs protocol to share disks into the VM guest that it doesn’t support read-only mode at all. This has now also been fixed by getting LXD to set up a read-only bind mount of the source directory on the host before sharing that with virtfs-proxy-helper (for 9p) and virtiofsd (for virtio-fs) respectively. This means we now take the “belt and braces” approach to enforcing readonly access for readonly disk shares, and do not only rely on the QEMU provided security features.
Also VM disk related, an issue that was causing the SCSI-ID (that is used to derive the disk device name inside the guest, e.g. /dev/sdb) of attached disk devices to fluctuate between boots has been fixed. The issue was caused by the use of the generated boot index to derive the SCSI-ID. This boot index number was changing for disk devices that did not explicitly specify a boot.priority setting. We now use a stable sorted list of devices when generating the boot index so that this doesn’t fluctuate any more.
A recently added safety feature that prevented using a host side network interface as the physical uplink parent for an OVN network if it had any IPs configured on it has been partially relaxed. We now only prevent this from happening if the parent interface is not a bridge. This is because it is valid to use a bridge interface (native or OVS) as the physical parent for an OVN network and because in this case the OVN network is connected to it via a bridge port it doesn’t matter if the bridge interface itself has IPs configured on it as OVN won’t disrupt the communication on that bridge.
The recently added cluster join token feature has had a small usability improvement, we now output the token from lxc cluster add on a new line so that it is easily selectable (to copy) using a triple-click.
LXC
The lxc-net script now supports the nftables firewall. Also some build errors on CentOS 7 have been fixed.
LXCFS
Support for virtualising /sys/devices/system/cpu/cpu* has been added.
Distrobuilder
Support has been added for both Rocky Linux, Springdale Linux and Amazon Linux distributions, which are all RHEL derived rebuilds.
A generic rootfs HTTP downloader called rootfs-http has been added to allow downloading premade rootfs images as the basis of distrobuilder built images.
A new --disable-overlay flag has been added that disables the use of overlayfs, and instead uses rsync for moving files around. Although overlayfs is faster, it causes issues on some setups (such as ZFS, which we already detect and automatically disable overlayfs use for) and this new flag allows users to rule that out as a cause of any issues they may be experiencing.
Dqlite (RAFT library)
Snapshot compression with lz4 has been introduced to avoid an issue that was occurring when a dqlite database was receiving lots of writes and had slow disk I/O causing periodic hangs for several seconds whilst the snapshot was written to disk. As well as compressing the snapshot, there’s an additional data integrity check when decompressing the compressed snapshot that in practice should allow dqlite to detect storage corruption. Members running compression are compatible with members running without compression, as the data over the wire hasn’t changed, this should deal with issues where cluster members are only updated one at a time. The compressed snapshot files are regular lz4 files (without the .lz4 extension though) that can be decompressed with the lz4 cli tool.
Dqlite (Go bindings)
A new optional argument has been added to the .dump command to allow specifying which database to dump.
Youtube channel
We’ve started a Youtube channel with live streams covering LXD releases and its use in the wider ecosystem.
You may want to give it a watch and/or subscribe for more content in the coming weeks.
Contribute to LXD
Ever wanted to contribute to LXD but not sure where to start?
We’ve recently gone through some effort to properly tag issues suitable for new contributors on Github: Easy issues for new contributors
Upcoming events
- Nothing to report this week
Ongoing projects
The list below is feature or refactoring work which will span several weeks/months and can’t be tied directly to a single Github issue or pull request.
- Distrobuilder Windows support
- Virtual networks in LXD
- Various kernel work
- Stable release work for LXC, LXCFS and LXD
Upstream changes
The items listed below are highlights of the work which happened upstream over the past week and which will be included in the next release.
LXD
- lxd/lxd: Record requestor as part of lifecycle events
- Add cluster time skew warning
- warnings: Add disabled AppArmor warning
--prettysupport forlxc monitorlifecycle events- README: Update IRC
- VM: Fix readonly disk shares
- Instance: Improve revert when device setup fails
- VM: Deduplicate the virtiofsd start and stop logic and simplify disk apparmor rules
- VM: Ensure bootindex ordering is stable and consistent
- VM: Create bind mount of config drive and instance disk devices for exporting via 9p and virtiofsd
- lxd/instance/qemu: Remove duplicate key
- lxc/publish: Fix ETag race condition
- Network: Relax physical network parent IP checks
- lxc cluster add shouldn’t have any alias
- doc: mention /var/snap/lxd/common/lxd for snap users
- lxd: print the join token on a separate line
LXC
LXCFS
Distrobuilder
- Add Rocky Linux
- main: Add systemd generator
- Add new distribution Springdale Linux
- main: Add --disable-overlay flag
- Add generic HTTP downloader
- main: Support source.URL template
- systemd: Refactor generator
- systemd-generator: Some small fixes
Dqlite (RAFT library)
Dqlite (database)
- Nothing to report this week
Dqlite (Go bindings)
Distribution work
This section is used to track the work done in downstream Linux distributions to ship the latest LXC, LXD and LXCFS as well as work to get various software to work properly inside containers.
Ubuntu
- Nothing to report this week
Snap
- Nothing to report this week