The LXD team is very excited to announce the release of LXD 3.13!
This is another very exciting LXD release, packed with useful features and a lot of bugfixes and performance improvements!
The latest addition to the LXD team, @tomp has been busy improving the LXD networking experience with quite a few new features and bugfixes already making it into this release.
We’ve also gotten all the plumbing needed for system call interception done and in place in this release, currently handling
mknod on supported systems.
Cluster users will enjoy this release too, thanks to scaling improvements, reducing the load on the leader a bit and improving container copies and migration, especially on CEPH clusters.
Enterprise users will like the addition of Role Based Access Control through the external Canonical RBAC service, making it possible to control permissions to individual projects on your LXD servers and assign roles to your users and groups.
And we’ve even managed to get quotas working for the
dir storage backend at last, thanks to the addition of filesystem project quotas in recent kernels.
Cluster: Improved heartbeat interval
In a LXD cluster, the current leader periodically sends a hearbeat to all other cluster members. The main purpose of this is to detect offline cluster members, marking them as offline in the database so that queries no longer block on them. A secondary use for those hearbeats is to refresh the list of database nodes.
Previously, this was done every 4s with all cluster members being contacted at the same time, resulting in spikes in CPU and network traffic, especially on the current cluster leader.
LXD 3.13 changes that by bumping the interval to 10s and by adding randomization to the timing of the hearbeats so that not all cluster members are contacted at the same time. Extra logic was also added to detect cluster members that get added during a hearbeat run.
Cluster Internal container copy
LXD 3.13 now properly implements one step container copies, similar to how you would normally copy a container on a standalone LXD instance. Prior to this, the client had to know whether to perform a copy (if staying on the same cluster member) or a migration (if going to another cluster member), this is now all done internally.
A side benefit of this fix is that all CEPH copies are now near instantaneous on clusters as those do not require any migration at all.
Initial syscall interception support
LXD 3.13 when combined with a 5.0 or higher kernel, as well as the very latest libseccomp and liblxc can now intercept and mediate system calls in userspace.
For this first pass, we have focused on
mknod, implementing a basic allow list of devices which can now be created by unprivileged containers.
It will take a little while before this feature can be commonly used as we will need an upstream release of both libseccomp and liblxc and are waiting for further improvements to the feature in the kernel too.
We will be building upon this capability to allow specific filesystems to be mounted inside unprivileged containers in the future as well as allow things like kernel module loading and more (all will require opt-in from the administrator).
Role Based Access Control (RBAC)
Users of the Canonical RBAC service can now integrate LXD with it.
LXD will register all its projects with RBAC, allowing administrators to assign roles to users/groups for specific projects or for the entire LXD instance.
Currently this includes the following permissions:
- Full administrative access to LXD
- Management of containers (creation, deletion, re-configuration, …)
- Operation of containers (start/stop/restart, exec, console, …)
- Management of images (creation, deletion, aliases, …)
- Management of profiles (creation, deletion, re-configuration, …)
- Management of the project itself (re-configuration)
- Read-only access (view everything tied to a project)
This gets us one step closer to being able to run a shared LXD cluster with unprivileged users being able to run containers on it without concerns of them escalating their privileges.
LXD can now make use of the recent implementation of
ipvlan in LXC.
When running a suitably recent version of LXC, IPVLAN can now be configured in LXD through a
- Setting the
- Setting the
parentproperty to the expected outgoing device
- For IPv4, setting
ipv4.addressto the desired address
- For IPv6, setting
ipv6.addressto the desired address
Here is an example of it in action:
stgraber@castiana:~$ lxc init ubuntu:18.04 ipvlan Creating ipvlan stgraber@castiana:~$ lxc config device add ipvlan eth0 nic nictype=ipvlan parent=wlan0 ipv4.address=172.17.0.100 ipv6.address=2001:470:b0f8:1000:1::100 Device eth0 added to ipvlan stgraber@castiana:~$ lxc start ipvlan stgraber@castiana:~$ lxc exec ipvlan bash root@ipvlan:~# ifconfig eth0: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1500 inet 172.17.0.100 netmask 255.255.255.255 broadcast 255.255.255.255 inet6 2001:470:b0f8:1000:1::100 prefixlen 128 scopeid 0x0<global> inet6 fe80::28:f800:12b:bdf8 prefixlen 64 scopeid 0x20<link> ether 00:28:f8:2b:bd:f8 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 5 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 root@ipvlan:~# ip -4 route show default dev eth0 root@ipvlan:~# ip -6 route show 2001:470:b0f8:1000:1::100 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default dev eth0 metric 1024 pref medium root@ipvlan:~# ping 22.214.171.124 PING 126.96.36.199 (188.8.131.52) 56(84) bytes of data. 64 bytes from 184.108.40.206: icmp_seq=1 ttl=57 time=14.4 ms --- 220.127.116.11 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 14.476/14.476/14.476/0.000 ms root@ipvlan:~# ping6 -n 2607:f8b0:400b:800::2004 PING 2607:f8b0:400b:800::2004(2607:f8b0:400b:800::2004) 56 data bytes 64 bytes from 2607:f8b0:400b:800::2004: icmp_seq=1 ttl=57 time=21.2 ms --- 2607:f8b0:400b:800::2004 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 21.245/21.245/21.245/0.000 ms root@ipvlan:~#
Quota support on
dir storage backend
Support for the
project quota feature of recent Linux kernels has been added.
When the backing filesystem for a
dir type storage pool is suitably configured, container quotas can now be set as with other storage backends and disk usage is also properly reported.
stgraber@castiana:~$ sudo truncate -s 10G /tmp/ext4.img stgraber@castiana:~$ sudo mkfs.ext4 /tmp/ext4.img mke2fs 1.44.6 (5-Mar-2019) Discarding device blocks: done Creating filesystem with 2621440 4k blocks and 655360 inodes Filesystem UUID: d8ab56d9-1e84-40ee-921a-c68c06ad6625 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 Allocating group tables: done Writing inode tables: done Creating journal (16384 blocks): done Writing superblocks and filesystem accounting information: done stgraber@castiana:~$ sudo tune2fs -O project -Q prjquota /tmp/ext4.img tune2fs 1.44.6 (5-Mar-2019) stgraber@castiana:~$ sudo mount -o prjquota /tmp/ext4.img /mnt/ stgraber@castiana:~$ sudo rmdir /mnt/lost+found/ stgraber@castiana:~$ lxc storage create mnt dir source=/mnt Storage pool mnt created stgraber@castiana:~$ lxc launch ubuntu:18.04 c1 -s mnt Creating c1 Starting c1 stgraber@castiana:~$ lxc exec c1 -- df -h Filesystem Size Used Avail Use% Mounted on /var/lib/lxd/storage-pools/mnt/containers/c1/rootfs 9.8G 742M 8.6G 8% / none 492K 0 492K 0% /dev udev 7.7G 0 7.7G 0% /dev/tty tmpfs 100K 0 100K 0% /dev/lxd tmpfs 100K 0 100K 0% /dev/.lxd-mounts tmpfs 7.8G 0 7.8G 0% /dev/shm tmpfs 7.8G 152K 7.8G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup stgraber@castiana:~$ lxc config device set c1 root size 1GB stgraber@castiana:~$ lxc exec c1 -- df -h Filesystem Size Used Avail Use% Mounted on /var/lib/lxd/storage-pools/mnt/containers/c1/rootfs 954M 706M 249M 74% / none 492K 0 492K 0% /dev udev 7.7G 0 7.7G 0% /dev/tty tmpfs 100K 0 100K 0% /dev/lxd tmpfs 100K 0 100K 0% /dev/.lxd-mounts tmpfs 7.8G 0 7.8G 0% /dev/shm tmpfs 7.8G 152K 7.8G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup stgraber@castiana:~$ lxc info c1 Name: c1 Location: none Remote: unix:// Architecture: x86_64 Created: 2019/05/09 16:09 UTC Status: Running Type: persistent Profiles: default Pid: 10096 Ips: eth0: inet 10.166.11.38 vethKM0DFY eth0: inet6 2001:470:b368:4242:216:3eff:fe4b:2c3 vethKM0DFY eth0: inet6 fe80::216:3eff:fe4b:2c3 vethKM0DFY lo: inet 127.0.0.1 lo: inet6 ::1 Resources: Processes: 24 Disk usage: root: 739.77MB CPU usage: CPU usage (in seconds): 7 Memory usage: Memory (current): 104.91MB Memory (peak): 229.67MB Network usage: lo: Bytes received: 1.23kB Bytes sent: 1.23kB Packets received: 12 Packets sent: 12 eth0: Bytes received: 480.35kB Bytes sent: 27.21kB Packets received: 332 Packets sent: 277
Routes on container NIC devices
ipv6.routes options on the
nic devices make it possible to tie a particular route to a specific container, making it follow the container as it’s moved between hosts.
This will usually be a better option than using the similarly named key on the network itself.
Configurable NAT source address
ipv6.nat.address properties on LXD networks now make it possible to override the outgoing IP address for a particular bridge.
LXC features exported in API
Similar to what was done in the previous release with kernel features, specific LXC features which LXD can use when present are now exported by the LXD API so that clients can check what advanced feature to expect on the target.
lxc_features: mount_injection_file: "true" network_gateway_device_route: "true" network_ipvlan: "true" network_l2proxy: "true" seccomp_notify: "true"
- client: Consider volumeOnly option when migrating
- client: Copy volume config and description
- client: Don’t crash on missing stdin
- client: Fix copy from snapshot
- client: Fix copying between two unix sockets
- doc: Adds missing packages to install guide
- doc: Correct host_name property
- doc: Update storage documentation
- i18n: Update translations from weblate
- lxc/copy: Don’t strip volatile keys on refresh
- lxc/utils: Updates progress to stop outputting if msg is longer than window
- lxd/api: Rename alias* commands to imageAlias*
- lxd/api: Rename apiProject* to project*
- lxd/api: Rename certificateFingerprint* to certficate*
- lxd/api: Rename operation functions for consistency
- lxd/api: Rename serverResources to api10Resources
- lxd/api: Rename snapshotHandler to containerSnapshotHandler
- lxd/api: Replace Command with APIEndpoint
- lxd/api: Sort API commands list
- lxd/candid: Cleanup config handling
- lxd/certificates: Make certificate add more robust
- lxd/certificates: Port to APIEndpoint
- lxd/cluster: Avoid panic in Gateway
- lxd/cluster: Fix race condition during join
- lxd/cluster: Port to APIEndpoint
- lxd/cluster: Use current time for hearbeat
- lxd/cluster: Workaround new raft logging
- lxd/containers: Avoid costly storage calls during snapshot
- lxd/containers: Change disable_ipv6=1 to accept_ra=0 on host side interface
- lxd/containers: Don’t fail on old libseccomp
- lxd/containers: Don’t needlessly mount snapshots
- lxd/containers: Early check for running container refresh
- lxd/containers: Fix bad operation type
- lxd/containers: Fix profile snapshot settings
- lxd/containers: Moves network limits to network up hook
- lxd/containers: Only run network up hook for nics that need it
- lxd/containers: Optimize snapshot retrieval
- lxd/containers: Port to APIEndpoint
- lxd/containers: Remove unused arg from network limits function
- lxd/containers: Speed up simple snapshot list
- lxd/daemon: Port to APIEndpoint
- lxd: Don’t allow remote access to internal API
- lxd: Fix volume migration with snapshots
- lxd: Have Authenticate return the protocol
- lxd: More reliably grab interface host name
- lxd: Port from HasApiExtension to LXCFeatures
- lxd: Rename parseAddr to proxyParseAddr
- lxd: Use idmap.Equals
- lxd/db: Fix substr handling for containers
- lxd/db: Parent filter for ContainerList
- lxd/db/profiles: Fix cross-project updates
- lxd/db: Properly handle unsetting keys
- lxd/event: Port to APIEndpoint
- lxd/images: Fix project handling on copy
- lxd/images: Fix simplestreams cache expiry
- lxd/images: Port to APIEndpoint
- lxd/images: Properly handle invalid protocols
- lxd/images: Replicate images to the right project
- lxd/internal: Port to APIEndpoint
- lxd/migration: Fix feature negotiation
- lxd/network: Filter leases by project
- lxd/network: Fix DNS records for projects
- lxd/network: Port to APIEndpoint
- lxd/operation: Port to APIEndpoint
- lxd/patches: Fix LVM VG name
- lxd/profiles: Optimize container updates
- lxd/profiles: Port to APIEndpoint
- lxd/projects: Port to APIEndpoint
- lxd/proxy: Correctly handle unix: path rewriting with empty bind=
- lxd/proxy: Don’t wrap string literal
- lxd/proxy: Fix goroutine leak
- lxd/proxy: Handle mnts for abstract unix sockets
- lxd/proxy: Make helpers static
- lxd/proxy: Make logfile close on exec
- lxd/proxy: Only attach to mntns for unix sockets
- lxd/proxy: Retry epoll on EINTR
- lxd/proxy: Use standard macros on exit
- lxd/proxy: Validate the addresses
- lxd/resource: Port to APIEndpoint
- lxd/storage: Don’t hardcode default project
- lxd/storage: Fix error message on differing maps
- lxd/storage: Handle XFS with leftover journal entries
- lxd/storage: Port to APIEndpoint
- lxd/storage/btrfs: Don’t make ro snapshots when unpriv
- lxd/storage/ceph: Don’t mix stderr with json
- lxd/storage/ceph: Fix snapshot of running containers
- lxd/storage/ceph: Fix snapshot of running xfs/btrfs
- lxd/storage/ceph: Fix UUID re-generation
- lxd/storage/ceph: Only rewrite UUID once
- lxd/sys: Cleanup State struct
- scripts/bash: Add bash completion for profile/container device get, set, unset
- shared: Add StringMapHasStringKey helper function
- shared: Fix $SNAP handling under new snappy
- shared: Fix Windows build
- shared/idmap: Add comparison function
- shared/netutils: Adapt to kernel changes
- shared/netutils: Add AbstractUnixReceiveFdData()
- shared/netutils: Export peer link id in getifaddrs
- shared/netutils: Handle SCM_CREDENTIALS when receiving fds
- shared/netutils: Move network cgo to shared/netutils
- shared/netutils: Move send/recv fd functions
- shared/network: Fix reporting of down interfaces
- shared/network: Get HostName field when possible
- shared/osarch: Add i586 to arch aliases
- tests: Extend migration tests
- tests: Handle built-in shiftfs
- tests: Updates config tests to use host_name for nic tests
Try it for yourself
This new LXD release is already available for you to try on our demo service.
The release tarballs can be found on our download page.