LXD 3.21 has been released

Introduction

The LXD team is very excited to announce the release of LXD 3.21!

This was another shorter two weeks development cycle for us, full of storage, VM and network bugfixes and refactoring.

On the feature side of things, we have a new easier way to attach network interfaces, clustering database configurations and a variety of virtual machine improvements.

Enjoy!

Highlights

New way to attach to LXD managed networks

When using a bridge network that’s directly managed by LXD (see lxc network list), you can now attach instances to it directly by using:

eth0:
    type: nic
    network: lxdbr0
    name: eth0

Or lxc config device add c1 eth0 nic network=lxdbr0 name=eth0

No need to set nictype: bridged or the parent: lxdbr0 property. Instead just set network to the name of the LXD network and you’re done.

In this mode, the bridge MTU is automatically inherited by the network interface, IPv4/IPv6 addresses are validated against the network’s configured subnets and the MAAS IPv4 and IPv6 subnets can now be configured through the network rather than the individual interfaces.

Ceph ported to new storage driver infrastructure

The Ceph storage driver is the latest and last storage driver to be ported to the new storage infrastructure. This should be an invisible change to users, but we would recommend users of Ceph on LXD try upgrading non-critical systems to 3.21 first and promptly report any issue related to storage.

With this last driver ported, we can now begin the work of removing all the old storage infrastructure from the LXD codebase, which should make things significantly easier to maintain going forward.

Clustering: Configurable number of active and standby database members

Two new configuration options have been added for clusters.

  • cluster.max_voters configures the expected number of active database cluster members (voting)
  • cluster.max_standby configures the expected number of standby database cluster members (non-voting)

The default configuration is for 3 voting members and 2 standby members. When a voting member goes down, a standby immediately gets promoted to voting and a spare idle member may then get promoted to standby.

Increasing the number of voting members will slow down the database performance as more members will need to agree on a change. Increasing the number of standby members will not affect database performance but will increase network load as more members will need to get the database binary stream.

Only the largest of clusters where loosing 2 or 3 members almost immediately is a possibility should consider increasing those default values.

VM: CPU pinning and topology

The limits.cpu key can now be set to specific CPU IDs just as is supported for containers. An example would be limits.cpu: 0,2 or limits.cpu: 0-3.

On top of pinning the VM’s virtual CPUs to those listed physical CPUs (or threads), LXD also attempts to match up the CPU topology of the VM.

In the example above, limits.cpu: 0,2, on a dual-core Intel system with hyper-threading, this refers to the first core and its hyper-thread. The resulting virtual machine will therefore be configured with a single socket, single core, hyper-threaded CPU and have both threads be pinned to match the physical hardware.

The same logic supports, multi-socket, multi-core systems with or without hyperthreading. So long as the CPUs listed in limits.cpu correspond to a realistic hardware configuration (same number of cores coming from each socket, same number of threads on all cores, …), then LXD will make the VM configuration match and will ensure that the pinning is done to match too.

If the requested configuration isn’t correct (doesn’t line up with hardware), LXD will fallback to providing a single socket, multi-core VM without hyperthreading, treating each of the list CPU IDs as a core. This obviously isn’t ideal and a warning will be logged to the LXD log when this happens.

VM: Network and storage optimizations

For improved network performance, LXD now uses vhost_net for its network devices.

On the storage front, discard is now enabled on our virtio-scsi drives, allowing for blocks to be discarded, shrinking the underlying file storage or allowing backing drives to better manage their blocks.

VM: Agent-less reporting of IPv6 addresses

Up until now, virtual-machine IP addresses were retrieved only from LXD’s DHCP server.
This works quite reliably for IPv4, but most IPv6 deployments use SLAAC which doesn’t come with a DHCPv6 lease and so wouldn’t be reported by LXD.

One way around this was to rely on the LXD agent as when it’s running inside the VM, it will be used to retrieve the network information for the virtual machine.

But for cases where that’s not possible, LXD now also looks at the IPv6 neighborhood records for any matching IPv6 addresses.

stgraber@castiana:~$ lxc list win10
+-------+---------+----------------------+----------------------------------------------+-----------------+-----------+
| NAME  |  STATE  |         IPV4         |                     IPV6                     |      TYPE       | SNAPSHOTS |
+-------+---------+----------------------+----------------------------------------------+-----------------+-----------+
| win10 | RUNNING | 10.166.11.118 (eth0) | 2001:470:b368:4242:9dff:908:98a9:c0c3 (eth0) | VIRTUAL-MACHINE | 0         |
+-------+---------+----------------------+----------------------------------------------+-----------------+-----------+

Complete changelog

Here is a complete list of all changes in this release:

Full commit list
  • lxd/migrate/container: Fixes migrate refresh final sync snapshot bug
  • lxd/migration/migration/volumes: Comment on Data property of VolumeSourceArgs
  • lxd/storage/drivers/driver/zfs/volumes: Explain use of volSrcArgs.Data for migration
  • lxd/instance/drivers/load: Pass copy of device config to device.Validate
  • lxd/device/nic/bridged: Updates use of network pkg functions
  • lxd/device/nic/bridged: Uses network.LoadByName to access n.HasDHCPvX() helpers
  • lxd/device: networkRandomDevName usage
  • lxd/network/network/load: Adds LoadByName function
  • lxd/network: Adds network type in network pkg
  • lxd/network/network/utils: Moves network utils from main pkg
  • lxd/instance/instance/utils: Removes NetworkUpdateStatic function link
  • lxd/instance/instance/utils: Adds more instance load functions
  • lxd/container: Removes instance load functions moved to instance pkg
  • container/lxc: network.UpdateDNSMasqStatic usage
  • lxd: instance.LoadNodeAll usage
  • lxd: instance.LoadByProject usage
  • lxd: instance.LoadByProjectAndName usage
  • lxd/device/device/utils/network: Updates network package usage
  • lxd/device/device/utils/network: Unexports some non-shared functions
  • lxd/network/utils: Removes network utils functions used by network type
  • lxd/networks/config: Removes networkFillAuto function
  • lxd/networks: Removes network type and networkLoadByName function
  • lxd/device: networkCreateVlanDeviceIfNeeded and networkRandomDevName usage
  • lxd: network package usage
  • test: static analysis of network pkg
  • lxd/instance/drivers/driver/qemu: network.GetLeaseAddresses usage
  • lxd/instance/instance/utils: Removes linked function NetworkGetLeaseAddresses var
  • lxd/network/network/utils: Adds GetMACSlice and GetLeaseAddresses functions
  • lxd/networks: Removes networkGetLeaseAddresses functions
  • lxd/networks/utils: Removes networkGetMacSlice function
  • lxd/instances: Fix URLs to use /1.0/instances
  • seccomp: make device number checking more robust
  • Define MS_LAZYTIME for compatibility with old glibc
  • lxd/vm: Use vhost_net
  • lxd/vm: Enable block discard
  • shared/archive: Fix out of space logic
  • lxd/vm: Set Documentation in systemd units
  • lxd/vm: Silence writeback warning for config drive
  • lxd/device/nic/bridged: Load br_netfilter kernel module when using IPv6 filtering
  • lxd/networks/configs: Adds maas.subnet.ipv{4,6} to allowed network keys
  • lxd: Device name quoting in device errors
  • lxd/device/nic: Adds network as valid nic property
  • lxd/networks: Uses HasDHCPv6 function and updates comment
  • lxd/network: Adds DHCP range parsing functions
  • lxd/device/nic/bridged: Updates to use network type DHCP ranges functions and types
  • lxd/device/nic/bridged: Adds support for network property
  • doc: Adds API extension for instance_nic_network
  • shared/version/api: Adds API extension for instance_nic_network
  • test/suites/container/devices/nic/bridged: Adds network property tests
  • doc: Adds network property to instance NIC bridged device
  • lxd/storage/zfs: Fix argument ordering
  • unix hotplug: skip devices without associated devpath or major/minor
  • lxd: Switches to simpler conn.WriteMessage function
  • lxd/storage/drivers: Add MountedRoot to Info
  • lxd/storage: Honor MountedRoot in pool actions
  • lxd/networks: Consider IPv6 neighborhood entries
  • lxd: Uses gorilla WriteJSON where possible
  • lxd/storage/drivers: Set MountedRoot option
  • lxd/main_checkfeature: add explicit _exit() even if it’s not needed
  • lxd/main_checkfeature: s/exit()/_exit()/g
  • cgo: export wait_for_pid() helper
  • lxd/main_checkfeature: close listener
  • lxd/main_checkfeature: don’t depend on kcmp header
  • lxd/device: Async CEPH unmap
  • lxd/storage/drivers/driver/lvm: Uses d.thinpoolName() rather than d.config[“lvm.thinpool_name”]
  • lxd/patches: setupStorageDriver usage
  • lxd/storage: Renames SetupStorageDriver to setupStorageDriver for consistency
  • lxd/storage/drivers/driver/zfs: Adds zfs kernel module load fail detection
  • lxd/daemon: setupStorageDriver usage
  • lxd/daemon: Comment consistency
  • lxd/storage/drivers/driver/lvm: Makes lvm.vg_name required for mounting
  • lxd/db/cluster/update: Adds updateFromV23 for ensuring lvm.vg_name key is set
  • lxd/db/cluster/update: Superfluous trailing whitespace
  • lxd/db/cluster/schema: v24 update
  • lxd/device/config/devices: Adds NICType function on Device type
  • lxd: Device.NICType usage
  • lxd/device/nic/bridged: Bans use of nictype when using network property
  • test: Updates nic bridged tests for NICType logic
  • lxd/network/network/utils: Fix network in use detection
  • lxd-agent/exec: Logs signal forwarding as info rather than error
  • lxd/container/exec: Only log finished mirror websocket when go routine exits
  • lxd/instance/drivers/driver/qemu: Fix go routine leak and hanging lxc clients
  • shared: Upper case first character of some debug messages
  • lxd/device/nic/bridged: Switches to dnsmasq.DHCPAllocatedIPs()
  • lxd/device/nic/bridged: Switches to dnsmasq.DHCPStaticIPs()
  • test/suites/container/devices/nic/bridged: Adds test to detect leaked filters
  • lxd/device/nic/bridged: Fixes bug that leaks ebtables filters
  • lxd/project: Adds InstanceParts() function for separating project prefixed Instance name
  • lxd/storage/load: Updates volIDFuncMake to use project.InstanceParts()
  • lxd/util: Fix IP/host tests on other distros
  • lxd/storage/drivers: Add Ceph driver
  • lxd: Use new storage code for Ceph clustering
  • Unlock when isLeader failure
  • lxd/storage/ceph: Function ordering and comments
  • lxd/storage/ceph: Properly handle os.Remove
  • lxd/storage/ceph: Comment consistency
  • lxd/storage/ceph: Set DirectIO
  • lxd/storage/ceph: Unwrap if statement
  • lxd/storage/ceph: Unwrap function signatures
  • lxd/storage/ceph: Rework MountVolume
  • lxd/patches: Re-run VM path creation
  • tests: Add ceph to list of new drivers
  • lxd/firewall: Moves iptables/xtables implementation into firewall/drivers package
  • Consider the default port when checking address overlap
  • lxd/firewall: Updates interface and loader for new pkg
  • lxd: firewall/drivers pkg usage
  • lxd/device/config/device/proxyaddress: Moves ProxyAddress type
  • lxd/main/forkproxy: Updates use of ProxyAddress type
  • lxd/device/proxy: Switches to use firewall.InstanceSetupProxyNAT()
  • lxd/firewall/firewall/interface: Reworks firewall interface
  • Re-disable clustering upgrade test
  • lxd: Fix error message when deleting storage pools
  • lxd/firewall/drivers/drivers/xtables: Implements xtables driver
  • lxd/network/network/utils: Adds UsesIPv4Firewall and UsesIPv6Firewall functions
  • lxd/device/nic/bridged: Switches to firewall.InstanceSetupBridgeFilter and InstanceClearBridgeFilter
  • lxd/network/network: firewall.NetworkSetupForwardingPolicy usage
  • lxd/network: firewall.NetworkSetupOutboundNAT usage
  • lxd/network: Updates firewall DHCP/DNS function usage
  • lxd/firewall/drivers/consts: Removes unused constants
  • lxd/network: Updates to use firewall helper functions
  • lxd/dnsmasq: Makes DHCPStaticIPs project aware
  • lxd/device/nic/bridged: dnsmasq.DHCPStaticIPs project usage
  • lxd/network/network/utils: dnsmasq.DHCPStaticIPs project usage
  • test: Removes old iptables package from static analysis
  • test: Fixes iptables rule leak in clustering test
  • shared: Add HostPathFollow
  • lxc/file: Follow symlinks on individual file transfers
  • lxd/container: Protect file push/pull from shift
  • Add cluster.n_voters and cluster.n_standby configuration keys
  • Load configuration keys when checking for n of voters/stand-by
  • doc/clustering.md: describe usage of clustering size config keys
  • Drive-by: fix check for degraded cluster
  • doc/server.md: add cluster.max_voters/max_standby
  • api: Add clustering_sizing extension
  • Revert “lxd/instance/drivers/driver/qemu: Fix go routine leak and hanging lxc clients”
  • lxd/instance: Move ParseCpuset
  • lxd/vm/qmp: Allow retrieving vCPU pids
  • lxd/vm: Implement CPU pinning
  • shared: get_poll_revents(): handle EAGAIN in addition to EINTR
  • lxc: send SIGTERM when there’s no controlling terminal
  • shared: Add Uint64InSlice
  • lxd/vm: Template sockets/cores/threads config
  • lxd/vm: Attempt to line up CPU topology
  • lxd init: Don’t allow empty strings for the cluster host name
  • node/config.go: Don’t allow wild card addresses for cluster.https_address
  • idmap:acl: don’t add but update the acls
  • shared/util: Tweak HostPathFollow to use readlink
  • lxc/file: Expand complex symlink chains
  • i18n: Update translations from weblate

Try it for yourself

This new LXD release is already available for you to try on our demo service.

Downloads

The release tarballs can be found on our download page.

Binary builds are also available for:

  • Linux: snap install lxd
  • MacOS: brew install lxc
  • Windows: choco install lxc
3 Likes

The LXD 3.21 snap is available in the candidate channel with promotion to stable expected on Monday or Tuesday.

What is best way to do an upgrade from 3.20, will a snap refresh do.

Right now, it’s only in candidate, so a snap refresh would need --candidate to cause a channel switch.

Once we release to stable on Monday/Tuesday, then yes, a simple snap refresh will do.

1 Like

Hello stgraber, I installed LXD in Ubuntu 18.04 (I did not know about snap at that time :cry:) the version that comes with Ubuntu 3.0 (as of today). However, I notice that with snap I have the option to install 3.20 & 3.21, what would be your recommendation for a less painful/successful migration from 3.0 to 3.20 in same box? I would like to stop using the version that comes with Ubuntu but I would prefer no to loose the current containers and configurations I have in place. But let me know if that is not possible, then I guess I will have to start from scratch…

Thank you in advance,

I did an upgrade from 3.20 to 3.1 on a small cluster and everything went well.
I did a snap refresh lxd --candidate
and then had to do systemctl reload snap.lxd & pkill -9 -f “lxd --logfile” several times before it started…
Lxd is hard to kill from memory
But when it came up it was fine.

snap install lxd && lxd.migrate

1 Like

Ohh wow this is pure awesomeness.

=== Source server
LXD version: 3.0.3
LXD PID: 1994
Resources:
  Containers: 13
  Images: 0
  Networks: 0
  Storage pools: 1

=== Destination server
LXD version: 3.20
LXD PID: 17975
Resources:
  Containers: 0
  Images: 0
  Networks: 0
  Storage pools: 0

The migration process will shut down all your containers then move your data to the destination LXD.
Once the data is moved, the destination LXD will start and apply any needed updates.
And finally your containers will be brought back to their previous state, completing the migration.

Are you ready to proceed (yes/no) [default=no]? 

Let me go with this and see.

Thank you for the prompt help.

I am sorry, this might sound like an obvious question, but, do I need to run this command below for the 13 containers I had running on the older version of lxd to function?

To migrate your existing client configuration, move ~/.config/lxc to ~/snap/lxd/current/.config/lxc

Thank you in advance,

No, it’s just a one time thing for your user account and only really relevant of you had custom remotes added, if not, you can skip that entirely.

1 Like

Sounds great,

I just noticed that one of my containers works just fine, the configuration of the network, packages installed, files I have saved on the container are all there, so all looks good.

I really appreciate the amazing work you have done for this project, love it more and more everyday. I hope with this version 3.20/3.21 the little annoying issue I was experiencing with the containers not able to ping other machines in my network got fixed, on times I would have to shutdown the LXD host and restart once or twice to have the network bridge in LXD to start working and allowing packets between the containers and the rest of the network.

Again, thank you so very much!

Promoted to stable, will rollout to most users within the next 24h.

After upgrading to 3.21 old containers cannot be deleted. They have been deleted from the database but still shows in list also zfs container delete do not work.

lxc@lxd1:~$ sudo lxc list --project testdev

+------------------------+----------+---------------------+------+-----------+-----------+-----------+
|          NAME          |  STATE   |        IPV4         | IPV6 |   TYPE    | SNAPSHOTS | LOCATION  |
+------------------------+----------+---------------------+------+-----------+-----------+-----------+
| test-api-dev-1          | RUNNING | ------------ (eth0) |      | CONTAINER | 0         | lxd1      |
+------------------------+----------+---------------------+------+-----------+-----------+-----------+
| test-api-dev-2020-01-29 | STOPPED |                     |      | CONTAINER | 0         | lxd2      |
+------------------------+----------+---------------------+------+-----------+-----------+-----------+
| test-api-devnew-1       | STOPPED |                     |      | CONTAINER | 0         | lxd2      |
+------------------------+----------+---------------------+------+-----------+-----------+-----------+
| test-app-dev-1          | STOPPED |                     |      | CONTAINER | 0         | lxd1      |
+------------------------+----------+---------------------+------+-----------+-----------+-----------+
| test-app-dev-2          | STOPPED |                     |      | CONTAINER | 0         | lxd3      |
+------------------------+----------+---------------------+------+-----------+-----------+-----------+
lxc@lxd1:~$ lxc delete test-app-dev-1
Error: not found
lxc@lxd1:~$ lxc delete icm-app-dev-1 --project testdev
Error: Failed to run: zfs destroy -r local/containers/testdev_icm-app-dev-1: cannot destroy 'local/containers/testdev_icm-app-dev-1': dataset is busy
lxc@lxd1:~$ sudo  zfs destroy -r local/containers/testdev_test-app-dev-1
cannot destroy 'local/containers/testdev_test-app-dev-1': dataset is busy
lxc@lxd1:~$ lxc start test-app-dev-1
Error: not found

Subsequently can start the containers again using lxc start test-app-dev-1 --project testdev.

Probably changes in storage driver might be causing it?

No, that’s a zfs error, the dataset is busy means that something’s currently using files from that ZFS dataset and it therefore cannot be deleted.

Your last lxc start is failing because you also didn’t specify the project there.

Thats the reason in my comment I mentioned I can start after correct command.

The problem is cannot delete those containers. What should be running when the containers are stopped. Also this happened after the upgrade, in earlier version the delete worked fine.

Indeed I tried to stop the lxd daemon and than try to delete the container using zfs command. It still gave an error.

This happened once the system upgrade to 3.21, and in the new release there is a storage driver update which might have caused it. I am using zfs pool in cluster.

Adding further information on this bug. This error is replicated again. Using zfs pool named local.

lxc@lxdserver1:~$ lxc stop test1
lxc@lxdserver1:~$ lxc copy test1 test2
lxc@lxdserver1:~$ lxc delete test1
lxdserver | FAILED | rc=1 >>
Error: Failed to run: zfs destroy -r local/containers/test1: cannot destroy 'local/containers/test1': dataset is busynon-zero return code

Still trying to fix I guess i am hitting this issue with zfsonlinux cannot destroy dataset: dataset is busy #1810

lxc@lxdserver1:~$ sudo grep test1 /proc/*/mounts
/proc/3585/mounts:local/containers/test1 /var/snap/lxd/common/lxd/storage-pools/local/containers/test1 zfs rw,xattr,posixacl 0 0
lxc@lxdserver1:~$ ps aux | grep 3585
root      3585  0.0  0.0 234844  1412 ?        Sl   Feb17   0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid

So the container cannot be deleted because lxcfs process is still trying to use it, giving an error dataset is busy.

Thats the reason in my comment I mentioned I can start after correct command.

The problem is cannot delete those containers. What should be running when the containers are off. Also this happened after the upgrade, in earlier version the delete worked fine.

I have a similar issue as well with deleting, exporting, renaming, taking snapshot for only some (4 out of 16)containers recently. It was working perfectly last time l used it, nothing has changed but the update to LXD is now version 3.21 I can start the container and stop it again!

| rhys-pg10 | STOPPED | | | CONTAINER | 0 |
±----------------±--------±----------------------±-----±----------±----------+
root@server01:~# lxc mv rhys-pg10 stuckContainer

Error: Rename instance: Failed to rename ‘/var/snap/lxd/common/lxd/storage-pools/lxd/containers/rhys-pg10’ to ‘/var/snap/lxd/common/lxd/storage-pools/lxd/containers/stuckContainer’: rename /var/snap/lxd/common/lxd/storage-pools/lxd/containers/rhys-pg10 /var/snap/lxd/common/lxd/storage-pools/lxd/containers/stuckContainer: device or resource busy
root@server01:~# lxc delete rhys-pg10
Error: Failed to run: zfs destroy -r lxd/containers/rhys-pg10: umount: /var/snap/lxd/common/shmounts/storage-pools/lxd/containers/rhys-pg10: mountpoint not found cannot unmount ‘/var/snap/lxd/common/shmounts/storage-pools/lxd/containers/rhys-pg10’: umount failed