Introduction
The LXD team is very excited to announce the release of LXD 5.12!
This release contains a LOT of fixes, especially around storage and instance migration.
On top of all those fixes, it also contains three small but rather useful new features.
Enjoy!
New features and highlights
Device wipe when creating storage pools
It’s now possible to instruct LXD to wipe the source device of a storage pool prior to creation. This should be used with extreme care as setting the wrong source value will cause the disk in question to have its header wiped clean by LXD.
This was added for use by MicroCloud where all of the disks are accounted for and this can be safely performed. For other use cases, you’ll want to make sure that there can’t possibly be useful data on the disk prior to using this.
To use it, just set source.wipe=true at storage pool creation time.
VM generation id
LXD now implements VM generation IDs.
This is a feature available on x86_64 VMs which has LXD expose a UUID that the guest can consume and compare with the UUID used at boot time. Any change in value will indicate that the guest has been restored to a former state and that sensitive cryptographic functions like the random number generator should be seeded again.
On LXD, the initial value of the generation ID will be the same as the VM UUID.
The generation ID will follow the same reset rules as the VM UUID with the exception that a snapshot restore will also trigger a generation ID reset whereas the instance UUID will be kept the same.
For users, there’s really nothing to do about this, it’s purely an additional security feature that the guest OS may or may not use. As a result, this is done by default for all LXD VMs on architectures that support it (currently x86_64 only).
VM block cache mode
A new disk device configuration option, io.cache has been introduced.
This option can be used to control the caching behavior for the disk in virtual machines.
The default value (none) will have LXD configure no caching and setup the use of Direct I/O when supported by the underlying storage pool. An alternative is to use writeback which will cache writes, flushing them to the backing pool when the guest sends a flush request. The last option is unsafe which works the same as writeback but ignores guest requests to flush the cache to disk.
Complete changelog
Here is a complete list of all changes in this release:
Full commit list
- lxd/storage/drivers/generic/vfs: Improve logging in genericVFSCreateVolumeFromMigration
- lxd/migrate/instance: Improve migrationSourceWs errors
- lxd/migrate/instance: Improve migrationSink errors and logging
- lxd/migrate/storage/volumes: Improve migrationSink errors
- lxd/operations: Improves operationWebsocketGet error
- lxd/instance/snapshot: Fix wrapping and quoting of errors in snapshotPost
- lxd/migrate: Improve errors in {migrationSourceWs,migrationSink}.Connect
- lxd/migrate/instance: Clean up operation on failure
- lxd/main/forkmigrate: Improve error in cmdForkmigrate
- lxd/instance/drivers/driver/common: Allow onStopOperationSetup to inherit Start operation
- lxd/instance/drivers/driver/common: Adds validateStartup function
- lxd/instance/drivers/driver/qemu: Call d.common.validateStartup from validateStartup
- lxd/instance/drivers: d.validateStartup usage
- lxd/instance: Removes unused SaveConfigFile functions
- lxd/migrate/instance: Update checkForPreDumpSupport to not depend on liblxc
- lxd/instance/instance/interface: Adds operation lock to CriuMigrationArgs
- lxd/instance/drivers/driver/lxc: Adds reverter to initLXC
- lxd/instance/drivers/driver/lxc: Call d.UpdateBackupFile from startCommon
- lxd/instance/drivers/driver/lxc: Comment endings
- lxd/instance/drivers/driver/lxc: Simplify ctx var creation in Start
- lxd/instance/drivers/driver/lxc: Rework the way Start calls Migrate when doing stateful start
- lxd/instance/drivers/driver/lxc: Rework Stop to fix stateful stop
- lxd/instance/drivers/driver/lxc: Rework Snapshot to fix stateful snapshots
- lxd/instance/drivers/driver/lxc: Cleanup Restore
- lxd/instance/drivers/driver/lxc: Removes unused liblxc.MIGRATE_FEATURE_CHECK check
- lxd/instance/drivers/driver/lxc: Remove UpdateBackupFile call from Migrate when restoring stateful checkpoint
- lxd/instance/drivers/driver/lxc: Fix liblxc initialisation in Migrate
- client: Don’t indicate live migration when copying snapshots in CopyInstanceSnapshot
- test: Update CRIU tests
- lxd/instance: Improve stateful copy error message
- Move C and therefore CGO out of shared package
- doc: fix in documentation for
physicalNIC with thenetworkoption - doc: add a page about troubleshooting failing instances
- doc/devices/disk: add video link and clarify content
- doc/devices/unix-*: add link to YouTube video
- doc/devices/unix-*: add title for YouTube video
- Move socket to linux
- doc/storage: Document zfs.block_mode
- lxd/instance/drivers/driver/lxc: Prevent panic in cgroup if liblxc not initialized
- lxd/instance/drivers/driver/lxc: Adds is running check to Metrics
- lxd/api/metrics: Detect and ignore ErrInstanceIsStopped error from inst.Metrics()
- lxd/instance/drivers/driver/lxc: Only intialise cgroup reader/writer when needed
- lxc/query: Also handle json UnmarshalTypeError
- lxc/query: Validate actions
- i18n: Update translation templates
- lxd/device/nic/ipvlan: Configure host-side routes and neighbour proxy entries with LXD
- lxd/instances: Fix incorrect alias path in instanceSFTPCmd
- lxd/instances/get: Correct comment terminology and endings
- lxd/instances/get: Use if/else if for clarity
- lxd/instances/get: Lose unnecessary recursionStr var
- lxd/instances/get: Fix filtering in non-recursive mode
- lxd/instance/filter: Removes unused function Filter
- doc/clustering: add link to scriptlet specification
- doc/storage: add link to ZFS block mode spec
- doc/projects: add link to projects tutorial
- lxd/db/instances: Add Instance type and rework GetProjectAndInstanceNamesByNodeAddress into GetInstancesByMemberAddress to return them
- lxd/db/instances: Remove unused GetProjectInstanceToNodeMap function
- lxd/instances/get: Reworks doInstancesGet to use tx.GetInstancesByMemberAddress
- lxd/instances: Update instanceLoadNodeProjectAll to accept a context
- lxd/scriptlet/starlark: No need to check all map key types
- lxd/scriptlet/starlark: Use predictable ordering of Starlark dict keys
- lxd/instances/get: Limit the scope of localInstancesByID
- lxd/instances/get: Keep recursion and clauses logic together in doInstancesGet
- lxd/instances/get: Sort the result by project and then name in doInstancesGet
- lxd/instances/get: Simplify non-recursive URL output in doInstancesGet
- doc/explanation/clustering: Show use of fail() in instance placement scriptlet
- lxd/storage/backend/lxd: Pass volume config into SetVolumeQuota in SetInstanceQuota
- lxd/storage/drivers/driver/ceph/volumes: Log volume name during mount
- lxd/storage/drivers/driver/lvm/volumes: Log volume name during mount
- lxd/storage/drivers/zfs: don’t force atime to on
- lxd/storage/drivers/driver/zfs/volumes: Adds activateVolume and deactiveVolume functions
- lxd/storage/drivers/driver/zfs/volumes: Improve mount and unmount logging consistency
- lxc/storage/drivers/driver/zfs/volumes: Only call InUse once during SetVolumeQuota
- lxd/storage/drivers/driver/zfs/volumes: Fix block mode filesystem volume resize in SetVolumeQuota
- test: Update zfs block mode tests
- lxd-agent: Retry getting vsock client
- test: test overlapping addresses between proxy device and network forward
- instance: Check that there is no collisions between
proxydevice and a network fwd listenAddr - refactor: create
ProxyParseAddras part of thenetworkpkg - refactor: update
ProxyParseAddrreferences - refactor: remove old
ProxyParseAddrfunc indevicepkg - doc/debugging: remove information about
lxd --debug - doc/troubleshooting: move example for instance troubleshooting
- lxd/ip/link: Add support for applying Link setings atomically at Add time
- lxd-agent/network: ip.Link usage in reconfigureNetworkInterfaces
- lxd/device/device/utils/network: ip.Link usage in NetworkSetDevMTU
- lxd/device/device/utils/network: ip.Link usage in networkSRIOVSetupContainerVFNIC
- lxd/device/infiniband/physical: ip.Link usage in Start
- lxd/device/infiniband/sriov: ip.Link usage in startContainer
- lxd/device/nic/macvlan: ip.Link usage in Start
- lxd/device/nic/physical: ip.Link usage in Start
- lxd/network/driver/bridge: ip.Link usage for setting MTU
- lxd/network/driver/physical: ip.Link usage for setting MTU
- lxd/ip/link/veth: Restructure Veth type to replace PeerName with Peer Link field
- lxd/network/driver/ovn: ip.Link usage for setting MTU
- lxd/network/driver/ovn: ip.Veth Peer usage
- lxd/device/device/utils/network: Reworks networkCreateVethPair to use ip.Veth settings
- lxd/device/nic/macvlan: Update macvlan to apply settings to host interface atomically
- lxd/device/device/utils/network: Updates networkCreateTap to support queue.tx.length setting
- lxd/network/openvswitch/ovs: Update BridgeAdd to accept MAC and MTU settings
- lxd/network/driver/ovn: ovs.BridgeAdd usage
- lxd/network/driver/bridge: Create and apply bridge interface settings atomically
- lxd: Indent swagger docs to indicate preformatted blocks.
- lxd/sys: Updates comments to ensure they are properly gofmt’d.
- lxd/storage: Updates comments to ensure they are properly gofmt’d.
- lxd/storage/drivers: Updates comments to ensure they are properly gofmt’d.
- lxd/node: Updates comments to ensure they are properly gofmt’d.
- lxd/firewall/drivers: Updates comments to ensure they are properly gofmt’d.
- lxd/endpoints: Updates comments to ensure they are properly gofmt’d.
- lxd/db/schema: Updates comments to ensure they are properly gofmt’d.
- lxd/db/operationtype: Updates comments to ensure they are properly gofmt’d.
- lxd/db: Updates comments to ensure they are properly gofmt’d.
- lxd/ip/link: Update SetAddress to take net.HardwareAddr
- lxd/device: link.SetAddress usage
- lxd/network/driver/bridge: bridge.SetAddress usage
- lxd/storage/backend/lxd: No need to activate VM’s block volume in UpdateInstanceBackupFile
- lxd/device/none: Allow hot plugging for VMs
- lxd/instance/drivers/driver/qemu: Don’t collect metrics unless VM is running
- client/lxd: Handle error responses from LXD in rawWebsocket
- lxd-agent/server: Correct error handling in createCmd
- lxd/daemon: Correct error handling in createCmd
- lxd/storage/drivers: Match ceph keyring lookup logic
- lxd/storage/s3/types: Adds ErrorInvalidRequest constant
- lxd/api: Fixes storageBucketsServer to parse URL to properly extract bucket name from path
- lxd/storage/zfs: Don’t force the zpool creation
- api: storage_pool_source_wipe
- lxd/storage: Add support for source.wipe
- doc: add sphinx-copybutton extension
- doc: fix documentation warnings
- doc/cloud-init: move content around
- doc/cloud-init: add information from Advanced Guide on website
- instance: add support for VM generation ID
- test: test if the instance UUID is different after restoring its snapshot
- zfs: Regenerate UUID on optimized migration
- api: Add zfs_block_mode API extension
- lxd/main_interactive: Tweak LVM thin pool error
- doc/faq: provide the path to the client.crt for snap users too
- doc/authentication: provide the path to the client config for snap users too
- doc/debugging: mention path for snap users
- doc: use RFC5737 (reserved for doc) IPv4 instead of RFC1918
- Parameterize
digand explain origin of the port - doc/cloud-init: clean up existing documentation
- doc/faq: add a section explaining where lxc stores its config
- lxd/storage/zfs: Also pass -f to zpool create on source.wipe=true
- lxd/storage/drivers/zfs: remove redundant ZFS version detection
- lxd/storage/drivers/driver/lvm: Improve error in Create
- lxd/storage/drivers/driver/common: Adds filesystemFreeze function
- lxd/storage/drivers/driver/ceph/volumes: Updates CreateVolumeSnapshot to use filesystemFreeze
- lxd/storage/drivers/zfs/volumes: Freeze source filesystem volume when taking temporary snapshot in CreateVolumeFromCopy
- lxd: tweak do_move_forkmount()
- doc/installing: add link to side-loading tutorial
- doc/lxd.benchmark: fix instructions for delete command
- test: Use per-instance storage pool VG for clustering tests
- doc/devices/tpm: add video link and general explanation
- GitHub actions: cancel running tests
- shared/network: Comments
- lxd/apparmor: Don’t use deprecated io/ioutil
- lxd/instance/drivers/driver/lxc: Use internal migrate function
- lxd/instance/instance/interface: Split Migrate into MigrateSend and MigrateReceive
- lxd/instance/drivers/driver/qemu: Implement MigrateSend and MigrateReceive
- lxd/instance/drivers/driver/lxc: Implement MigrateSend and MigrateReceive
- lxd/migrate/instance: Use MigrateSend and MigrateReceive
- lxd/apparmor: fix AppArmor instance_qemu profile
- lxd/instance/instance/interface: Update MigrateSend to accept MigrateSendArgs
- lxd/instance/instance/utils: Move SnapshotToProtobuf into instance package
- lxd/instance/drivers/driver/lxc: Move container migration send logic into lxc driver
- lxd/instance/drivers/driver/qemu: Move VM migration send logic into qemu driver
- lxd/migrate/instance: Removes container migration logic from migrationSourceWs
- shared/network: Make sure subsequent reads after close block until next message in WebsocketIO
- lxd/instance/drivers/driver/qemu: Fix stateful error message in Snapshot
- lxd/instance/instance/utils: Adds SnapshotProtobufToInstanceArgs function
- lxd/migration/utils: Adds ControlResponse type
- lxd/migrate: migration.ControlResponse usage
- lxd/migrate/storage/volumes: migration.ControlResponse usage
- lxd/migrate: Removes unused migrationControlResponse type
- lxd/storage/migration: Removes unused snapshotProtobufToInstanceArgs
- lxd/instance/instance/interface: Update MigrateReceive to accept MigrateReceiveArgs
- lxd/instance/drivers/driver/lxc: Move container migration receive logic into lxc driver
- lxd/instance/drivers/driver/qemu: Adds start function
- lxd/instance/drivers/driver/qemu: Move VM migration receive logic into qemu driver
- lxd/storage: Removes unused resetContainerDiskIdmap function
- lxd/migrate/instance: Removes instance type specific migration receive logic from migrationSink
- lxd/instances/post: Persist operation for duration of createFromMigration
- lxd/device/disk: Use explicit args and pass charset to mkisofs in generateVMConfigDrive
- doc: enable an extension for terminal output
- doc: add terminal output where suitable
- test: Shorten boot.host_shutdown_timeout in clustering_evacuation tests
- Revert “instance: add support for VM generation ID”
- test: UUID should remain the same after back/snap restore
- test: UUID should be the same after instance move to other server
- lxd/migrate: Add send timeout to send function
- lxd/migrate/instance: Add error sending synchronisation in migrationSourceWs.Do
- lxd/instance/drivers/driver/qemu: Improve migration protocol error handling in MigrateReceive
- lxd/instance/drivers/driver/lxc: Improve migration protocol error handling in MigrateReceive
- lxd/migrate/instance: Removes control socket error response from migrationSink.Do
- lxd/instance/qemu: Fix SMP on s390x
- instance: Add
volatile.uuid.generationto VM instance config - lxd/devices: Updates deviceTaskBalance to check the container’s InitPID as running indicator
- test: check generation UUID with snap/backup restore
- lxd/instances/get: Fix clustered instances list
- doc: Add
volatile.uuid.generationandinstance_generation_idextension to doc - lxd/shared/version: Add the
instance_generation_idAPI extension - zfs: Ensure snapdev=visible is set correctly
- zfs: Add private mountVolumeSnapshot function
- zfs: Add “noload” mount option for ext4 mounts
- zfs: Always mount volume snapshots read-only
- zfs: Use mountVolumeSnapshot in readonlySnapshot
- zfs: Regenerate UUID on volume restore
- test: Test different filesystems for zfs block mode
- zfs: Properly clean up clone
- lxd/migrate: Unexport migrationSinkArgs
- lxd/instances/post: migrationSinkArgs usage
- lxd/migrate/instance: Add live field to logging
- lxd/instance/drivers: Clarify logging of control response sending
- lxd/migrate: Adds 30s send timeout to disconnect
- doc/devices/usb: add link to video and some explanation
- doc/images: link to Discourse post on LXD image servers
- lxd/instance/instance/interface: Adds Disconnect function to MigrateArgs
- lxd/migrate/instance: Setup Disconnect function
- lxd/instance/drivers: Disconnect all sockets on MigrateReceieve error
- lxd/instances/post: Fix incorrect redirection logic in instancesPost
- lxd/instance/drivers: Update comments for accuracy
- lxd/instances/post: Update comment for accuracy
- lxd/instance/instance/interface: Rename DataConn and LiveConn to FilesystemConn and StateConn
- lxd: Replace criuConn and dataConn references with stateConn and filesystemConn
- shared/api/migration: Define the migration secret names used in the migration protocol
- client: api.SecretName* usage
- lxc-to-lxd/utils: api.SecretName* usage
- lxd-agent/exec: api.SecretName* usage
- lxd-migrate/utils: api.SecretName* usage
- lxd: api.SecretName* usage
- lxd/instance/drivers/driver/lxc: Use correct constant CRIUType_NONE in comment
- lxd/migrate: Fix panic in migrationFields.send
- gomod: Adds golang.org/x/sync/errgroup
- lxd/instance/drivers/driver/lxc: Reworks MigrateSend to use errgroup and separate control monitor routine
- lxd/instance/drivers/driver/qemu: Reworks MigrateSend to use errgroup separate control monitor routine
- lxd/instance/drivers/driver/lxc: Rework MigrateReceive to use errgroup
- lxd/instance/drivers/driver/qemu: Rework MigrateReceive to use errgroup
- lxd/migrate/instance: Removes end control message exchange in migrationSourceWs.Do
- lxd/instance/qemu: Default to writeback for RBD
- api: disk_io_cache
- shared/simplestream: Don’t write zero length cache files in cachedDownload
- doc: enable the extension for config options
- doc: add a cheat sheet for config options
- lxd/device/disk: Add io.mode
- lxd/instance/qemu: Add support for io.cache
- doc: Add io.cache to disk devices
- lxd/storage/drivers/driver/zfs/volumes: Add cleanup in createVolumeFromMigrationOptimized
- lxd/instance/drivers/driver/lxc: Fix possible panic on error in MigrateSend
- lxd/storage/backend/lxd: Allow migrating running VMs stateless using rsync+dd when specifying --allow-inconsistent mode
- lxd/instance/instance/utils: Return error when failing to delete instance snapshot in DeleteSnapshots
- lxd/migrate/storage/volumes: Fix incorrect error message in newStorageMigrationSource
- zfs: Ensure source FS is synced before migrating
- lxd/db/instances: Updates UpdateInstanceNode to accept poolID
- lxd/instance/instance/interface: Adds ClusterSameNameMove hint field to MigrateReceiveArgs
- lxd/instance/post: Update internalClusterInstanceMovedPost to support deleting local instance volumes
- lxd/instance/post: Reworks instancePostClusteringMigrate to use full migration protocol
- lxd/instance/post: tx.UpdateInstanceNode usage
- lxd/instance/post: instancePostClusteringMigrate usage
- lxd/migrate: Adds cluster move hint to migration sink
- lxd/instances/post: Update createFromMigration to use cluster notification as instance move hint
- lxd/storage/backend/lxd: Update DeleteInstance to only check for existing snapshot volume DB records
- lxd/instance/drivers/driver/lxc: Update MigrateReceive to support cluster moves
- lxd/instance/drivers/driver/qemu: Update MigrateReceive to support cluster moves
- lxd/instance/post: instancePostClusteringMigrate
- test: Update clustering containers tests to check for live migration support
- lxd/migrate: Improve error messages
- lxd/migrate: Prevent reconnect attempts in migrationSourceWs.Connect and migrationSink.Connect
- lxd/migrate: Prevent panics if allConnected is closed multiple times by using a Canceller
- lxd/instance/drivers/driver/qemu: Improve operation cleanup and error reporting
- lxd/instance/drivers/qmp/command: Differentiate errors
- lxd/network: disable dnsmasq negative query cache
- Documentation: fixed typo
- gomod: Update dependencies
- i18n: Update translations from weblate
Try it for yourself
This new LXD release is already available for you to try on our demo service.
Downloads
The release tarballs can be found on our download page.
Binary builds are also available for:
- Linux: snap install lxd
- MacOS: brew install lxc
- Windows: choco install lxc