Introduction
The LXD team is very excited to announce the release of LXD 4.23!
This is a very busy release with four major features and quite a lot more smaller features, not to mention a lot of bug fixes and performance improvements.
Enjoy!
New features and highlights
lxd-migrate replaces lxd-p2c
lxd-p2c which allowed for importing an existing system or filesystem tree into a LXD container using the migration API has been replaced by lxd-migrate.
The new tool takes what lxd-p2c supported and adds in:
- Support for migrating disk images to LXD virtual machine
- Support for more authentication methods
- More fine grained overrides of network, storage, profiles and other configuration options
It otherwise remains a tool which can be statically built and only relies on rsync being present on the source system. The transfer is done through LXD’s migration API and so can be done against any LXD server, local or remote.
Specification: [LXD] Unified P2C/P2V migration tool
Token based authentication
As part of our effort to strength LXD’s API security, it is now possible to issue one-time token that allows a new client to interact with LXD. This is a much better alternative to using a shared trust password and is easier to handle than manually sending certificates around.
Similar to what was done with lxc cluster add, it is now possible to use lxc config trust add to issue a new token. This token includes connection information, the server certificate and a one time secret.
stgraber@castiana:~$ lxc config trust list
+------+------+-------------+-------------+------------+-------------+
| TYPE | NAME | COMMON NAME | FINGERPRINT | ISSUE DATE | EXPIRY DATE |
+------+------+-------------+-------------+------------+-------------+
stgraber@castiana:~$ lxc config trust add
Please provide client name: castiana
Client castiana certificate add token: eyJjbGllbnRfbmFtZSI6ImNhc3RpYW5hIiwiZmluZ2VycHJpbnQiOiI1N2FhYjFkMjNhMGRlODdjZmQxNzkwNzNkMDVlN2U5OGIyY2U2ZjRmNTM1NjVkYzUzOTY1MjQ1MzRkNWU1NjM0IiwiYWRkcmVzc2VzIjpbIjE3Mi4xNy4wLjE0MTo4NDQzIiwiWzI2MDI6ZmM2MjpiOjEwMDA6NDNhNjo2NTJlOmZlZjI6ZTg4Y10 6ODQ0MyIsIjEwLjEyOC4xOTIuMTo4NDQzIiwiW2ZkNDI6YWU1Zjo5OGFiOmE4MTY6OjFdOjg0NDMiXSwic2VjcmV0IjoiYjVmMzQwZmJlN2IxOWM3M2U2MDFjZTJkYTc4YzNhMTlhZDgwY2RmZDExZDRkOTA1YTg0ODE2MTE5NWI3YzIwNSJ9
stgraber@castiana:~$ lxc remote add my-server eyJjbGllbnRfbmFtZSI6ImNhc3RpYW5hIiwiZmluZ2VycHJpbnQiOiI1N2FhYjFkMjNhMGRlODdjZmQxNzkwNzNkMDVlN2U5OGIyY2U2ZjRmNTM1NjVkYzUzOTY1MjQ1MzRkNWU1NjM0IiwiYWRkcmVzc2VzIjpbIjE3Mi4xNy4wLjE0MTo4NDQzIiwiWzI2MDI6ZmM2MjpiOjEwMDA6NDNhNjo2NTJlOmZlZjI6ZTg4Y106ODQ0MyIsIjEwLjEyOC4xOTIuMTo4NDQzIiwiW2ZkNDI6YWU1Zjo5OGFiOmE4MTY6OjFdOjg0NDMiXSwic2VjcmV0IjoiYjVmMzQwZmJlN2IxOWM3M2U2MDFjZTJkYTc4YzNhMTlhZDgwY2RmZDExZDRkOTA1YTg0ODE2MTE5NWI3YzIwNSJ9
stgraber@castiana:~$ lxc list my-server:
+-----------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-----------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| lxd-build | RUNNING | 10.128.192.40 (eth0) | fd42:ae5f:98ab:a816:216:3eff:fe28:c0a6 (eth0) | CONTAINER | 0 |
+-----------+---------+----------------------+-----------------------------------------------+-----------+-----------+
stgraber@castiana:~$ lxc config trust list
+--------+----------+------------------+--------------+-------------------------------+-------------------------------+
| TYPE | NAME | COMMON NAME | FINGERPRINT | ISSUE DATE | EXPIRY DATE |
+--------+----------+------------------+--------------+-------------------------------+-------------------------------+
| client | castiana | UNKNOWN@castiana | 6f5a027652fb | Nov 16, 2021 at 12:27am (UTC) | Nov 14, 2031 at 12:27am (UTC) |
+--------+----------+------------------+--------------+-------------------------------+-------------------------------+
You’ll note that the client never had to provide the server address, validate the server fingerprint or provide a password as all of that is part of the token. A token can be restricted to a set of project at the time of creation, making this ideal for enrolling new users into a shared environment.
Specification: [LXD] Token based remote connection
Documentation: Remote API authentication - LXD documentation
Custom DNS records in network zones
Network zones were first introduced as a way to easily get auto-generated forward and reverse DNS records for LXD instances.
Now it’s also possible to manually add additional DNS records to those zones.
This is done through lxc network zone record or the matching API.
Those records can be particularly useful if doing DNS validation for services like Let’s Encrypt or when performing domain validation with some kind of 3rd party service.
Specification: [LXD] Custom DNS records in network zones
Documentation: Network Zones configuration - LXD documentation
Image requirements
LXD image can now make use of special properties to indicate requirements to LXD.
The initial implementation comes with two of those:
requirements.cgrouprequirements.secureboot
The former can be used to tell LXD that a particular image requires some kind of cgroup tree.
This currently only supports v1 as a value which will have LXD check that the host system still supports CGroupV1 (a common issue when running distributions such as CentOS 7 or Ubuntu 16.04 on a modern system which relies on cgroup2).
The latter can be set to false to indicate that the image cannot boot when secureboot is enabled.
In both cases, LXD will perform those checks on instance start and fail an error should the requirements not be met.
stgraber@castiana:~$ lxc launch images:centos/7 c1
Creating c1
Starting c1
Error: The image used by this instance requires a CGroupV1 host system
Documentation: https://linuxcontainers.org/lxd/docs/master/image-handling/#special-image-properties
Network ACL log access (OVN)
When using network ACLs on an OVN network, it is possible to set the rule state to logged so that any traffic hitting the rule causes a log entry.
Those log entries show up in the OVN log of the system running the instance affected.
Unfortunately, the OVN log isn’t always super readable, not to mention, unprivileged users will not have access to it.
That’s why we’re now adding an API and matching CLI to retrieve the log entries related to a particular ACL.
root@abydos:~# lxc network acl show-log unifi
{"time":"2022-02-12T06:01:26Z","proto":"tcp","src":"45.45.148.253","dst":"45.45.148.3","src_port":"33556","dst_port":"22","action":"reject"}
{"time":"2022-02-12T06:01:26Z","proto":"tcp6","src":"2602:fc62:b:1000:43a6:652e:fef2:e88c","dst":"2602:fc62:a:1::3","src_port":"52590","dst_port":"22","action":"reject"}
Specification: [LXD] Network ACL logging
Network state support for OVN
The network state API has now been made to work with OVN.
Concretely, this means that lxc network info now works against an OVN network.
VLAN ranges in vlan.tagged
The vlan.tagged configuration option always supported multiple VLANs as a comma separated list.
While this worked well for most environments, those dealing with a large number of VLANs tend to be more used to ranges.
As a result, LXD now supports using VLAN ID ranges by setting vlan.tagged to something like 10,50,1000-2000.
Server side filtering of storage volumes
Joining the existing instances and images APIs, the storage-volumes API now supports server-side filtering. It’s possible to filter on volume name, type, configuration and have the API return snapshots or not.
Documentation: REST API - LXD documentation
zfs.export storage pool configuration
A new zfs.export storage pool configuration option was introduced to control the ZFS pool export behavior on LXD shutdown.
By default, LXD will call zpool export on any storage pool that it fully owns, whether it’s created on a loop disk or using a dedicated disk/partition.
With this option, it’s possible to have LXD keep the zpool loaded in such cases.
zfs.reserve_space storage volume configuration
Another ZFs configuration addition is zfs.reserve_space which can be set on a volume and will have ZFS guarantee that the allocated disk space is available to that instance.
This can be particularly useful for critical services which you cannot afford to have run out of disk space should other instances on the system run you out of space.
zfs.blocksize storage volume configuration
One last ZFS configuration option is zfs.blocksize which can also be set on storage volumes and controls the block size or record size used. Tweaking this value can be quite helpful to either limit waste when dealing with a lot of very small files or to improve throughput when dealing with very large files.
@never option for snapshot schedule
Lastly, a new snapshot scheduling option @never was introduced.
This can be used to break inheritance of snapshots.schedule.
If snapshots.schedule is set in a profile and you want one instance using that profile to not perform snapshots, then setting snapshots.schedule to @never on the instance itself will do the trick.
Complete changelog
Here is a complete list of all changes in this release:
Full commit list
- doc: fix language in SECURITY.md
- lxd/dnsmasq: Append device name to dnsmasq entry file name.
- Take extra precautions when calling tar
- lxd: Applies patch to add device names to dnsmasq files.
- test: Updates test suite to append device names to dnsmasq files.
- doc: Adds note about newuidmap to machine setup.
- lxd/instances: Don’t check limits on restore
- tests: Validate snapshots on limited projects
- test: Fixes number of expected NAT rules when using xtables driver.
- lxc/config/file: allow overriding default remote by env variable
- doc/environment: add new LXC_REMOTE variable
- test: Ensure ebtables is legacy when driver is xtables.
- client: Addd GetInstanceFull
- lxc/info: Port to using GetInstanceFull
- lxc/list: Add InstanceFull shortcut
- client: Add WithContext
- client: Add context-aware connection functions
- lxd/instance: Allow stopping during evacuation
- lxd/cluster: Don’t start the instance after live-migration
- lxd/cluster: Add evacuateClusterSetState
- lxd/cluster: Use instance architecture instead of cluster member
- lxd/cluster: Use evacuateClusterSetState
- lxd/cluster: Fix incorrect reverter
- lxd/cluster: Fix live-migrate on restore
- lxd/instance/qemu: Extend live updatable config keys
- lxd: Refresh all dnsmasq hosts during patch.
- lxd/instance/drivers: Adds getRootDiskDevice function to common driver.
- lxd/instance/drivers: Performs disk size check on stateful startup.
- doc: add a security summary and include it where needed
- doc: add an authentication section
- doc: clean up security documentation
- doc: add styling for abbreviations
- doc: fix loading of substitutions
- Growing a loop backed btrfs pool fix snap path
- doc: use absolute URL for Swagger
- doc: swagger: hide link to yaml file
- lxd/db: Import ordering
- lxd/db: Fix volume sequence regression
- doc: clean up links in README
- doc: move missing content from SUPPORT.md to README.md
- doc: add a support page and include it
- doc: move SUPPORT.md to .github folder
- doc: add a link to the Code of Conduct
- doc: fix link to support documentation
- doc: fix broken links
- lxd/network: Adds State function to interace with common implementation.
- lxd: Call the networks ‘State’ function if it can be loaded.
- shared/validate: Introduce ParseNetworkVLANRange
- lxd/device: Introduce networkVLANListExpand
- lxd/device/nic: Enable VLAN ranges in vlan.tagged
- tests: Add tests for VLAN ranges
- doc/instances: Mention support for ranges in vlan.tagged
- lxd/device/nic: Fix VLAN range validation
- shared/validate: Align ParseNetworkVLANRange with ParseUint32Range
- lxd/device: Update for ParseNetworkVLANRange change
- lxd/storage/btrfs: Add volume delete shortcut
- shared/api: Fix typo
- doc/rest-api: Refresh swagger YAML
- client: Fix bad arg naming in zone functions
- doc: enable automatically generated anchors
- lxd/db: Fix storage_volumes sequence again
- lxd: Use projectParam function in networkStateGet.
- shared/api: Adds OVN network state to api response.
- lxd/network/openvswitch: Adds methods for extracting OVN data.
- lxd/network: Implements ‘State’ for the ovn driver.
- api: Add network_state_ovn extension.
- doc/rest-api: Update swagger YAML.
- lxd/network/acl: Port to using fmt.Errorf
- lxd/network/zone: Port to using fmt.Errorf
- lxd/network/openvswitch: Port to using fmt.Errorf
- lxd/network: Port to using fmt.Errorf
- lxd/instance/lxc: Use absolute rootfs
- lxd: Add server-side API filtering for storage volumes
- api: Add storage_volume_api_filtering extension
- doc/rest-api: Refresh swagger YAML
- doc: Update rest api filtering doc
- doc: kernel 5.15+ have sane value for net.core.bpf_jit_limit
- api: Add image_restrictions extension
- shared/simplestreams: Support image reqs metadata
- doc: Add notion of image reqs
- shared/util: Add IsFalse
- lxd/drivers: Add secureboot & cgroup image reqs
- doc: Add secureboot and cgroup image reqs
- lxd: Add zfs.export
- lxd/storage/drivers: Support zfs.export
- doc/storage: Add zfs.export
- api: Add storage_zfs_export extension
- lxd/instance/qemu: Enable HyperV flags on x86_64
- doc/storage: Ceph supports quotas
- tests: Fix failure when shiftfs is skipped
- lxd/instance/lxc: Fix mount injection on VFS idmap
- lxd-agent: Fix bad network metric
- lxd/endpoints/listeners: Add exportable listeners package
- lxd/endpoints/starttls: Remove TLS methods from endpoints package
- lxd/ucred/ucred: Use new listeners package
- lxd/instance/qemu: Restrict HyperV flags to 5.10+
- lxd/db: Fix stoarge_volumes sequence (again)
- lxd/dns: Better handle errors
- api: network_dns_records
- shared/api: Add network zone record structs
- doc/rest-api: Refresh swagger YAML
- client: Add network zone records
- lxc/network_zone: Add record sub-command
- i18n: Update translation templates
- lxd/db: Add the networks_zones_records tables
- lxd/db: Add network zone records DB functions
- lxd/lifecycle: Add network zone record events
- lxd/network/zone: Add record functions
- lxd/network: Add zone records API
- lxd/network/zone: Extend template for TTL
- lxd/network/zone: Add extra records to DNS
- tests: Add tests for network zones records
- doc/network-zones: Add section on custom records
- doc: Fix typo in ZFS storage
- api: storage_zfs_reserve_space
- lxd/storage: Add zfs.reserve_space
- doc/storage: Add zfs.reserve_space
- scripts: Add zfs.reserve_space to completion
- lxd/storage/drivers: Sets RunningCopyFreeze to true.
- doc: add a target to serve the rendered docs
- doc: add a doc README
- doc: exclude README.md from doc build
- lxd: Only patch dnsmasq for networks in the db.
- lxd/storage/drivers: Factors out fast snapshot logic from volume backup.
- lxd/storage/drivers: Uses fast snapshot as source of migration.
- lxd/storage/drivers: Factors out fast snapshot logic from volume backup.
- lxd/storage/drivers: Uses fast snapshot as source of migration.
- lxd/storage: Freezes instances during migration.
- lxc/exec: Don’t terminate on SIGWINCH
- lxd/events: Increase websocket pings to 10s
- lxd/console: Fix error wrapping
- lxc/console: Properly handle GUI exitting
- lxc/console: Fix typo
- lxd/storage/drivers: Fixes reverter usage.
- lxd: Restart networks when enabling clustering
- lxd/network/bridge: Skip HandleHeartbeat on missing forkdns
- api: network_acl_log
- client: Add GetNetworkACLLogfile
- lxc/network_acl: Add show-log
- i18n: Update translation templates
- lxd/network/acl: Add log endpoint
- doc/rest-api: Refresh swagger YAML
- lxd/storage/drivers: Add support for zfs.blocksize config option
- lxd: Add validation for volume.zfs.blocksize
- doc: Add zfs.blocksize
- api: Add storage_zfs_blocksize extension
- doc: quick cleanup of FAQ
- api: metrics_cpu_seconds
- lxd/metrics: Convert to using float64
- lxd/metrics: lxd_cpu_seconds_total is in seconds, not ms
- lxd/storage: Expose GetVolume
- lxd/instance/lxc: Fix filesystem metrics
- lxd/db: Refactor storage pool used by to get info on all nodes.
- lxd: Get storage pool used-by info from all nodes if target is unset.
- lxc/storage: Parse and include node in used-by info.
- Rename lxd-p2c to lxd-migrate
- Makefile: s/lxd-p2c/lxd-migrate/
- test/suites: s/lxd-p2c/lxd-migrate/
- .github/workflows: s/lxd-p2c/lxd-migrate/
- .gitignore: s/lxd-p2c/lxd-migrate/
- lxd/device: Removes VM from list of supported instance types.
- doc: Clarify multiple GPU device passthrough for VMs.
- doc/metrics: switch to ECDSA with longer validity
- doc/metrics: tune scrape_interval to deal with default caching done by LXD
- doc/metrics: alpha sort prometheus.yml snippet
- lxd/endpoints: Implements an io.Writer to skip certain input.
- lxd/endpoints: Adds tests for the networkServerErrorLogWriter.
- lxd/endpoints: Sets the network server logger when proxies are updated.
- lxd/instance/lxc: Fix missing fs metrics on bind-mount
- lxd/operations: Add ExtendMetadata
- lxd/exec: Use ExtendMetadata
- shared/api: Fix incorrect image aliases example
- doc/rest-api: Refresh swagger YAML
- doc: Add basic instance exec information
- go.mod: use github.com/mdlayher/vsock@v1.0.0
- lxd/instance/metrics: Fix incorrect memory metrics
- lxd/db/generate/db/stmt: Add ‘order’ tag
- lxd/db/instance/profiles: Use ‘order’ tag to order queries by apply order
- lxd/db/instance/profiles.mapper: Update generated code
- lxd/metrics: Invert condition as all existing metrics have labels
- lxd/instance/lxc: Cast statfs.Bsize only once
- lxd/instance/lxc: Only convert CPU ID once
- lxd/instance/lxc: Cache labels in for loops
- api: instance_snapshot_never
- doc/instances: Add @never to snapshots.schedule
- lxd: Add @never to snapshots.schedule
- lxd/instance/qemu: Fix live update logic
- lxd/instance/qemu: Fix agent-less memory metrics
- lxd/instance/qemu/qmp: Remove GetMemoryStats
- lxd/firewall/drivers/drivers/xtables: Don’t attempt IPv6 RP filter if not enabled
- lxd/network/driver/ovn: Don’t attempt to configure IPv6 setting is not enabled
- doc/instances: Fix bridged NIC ipv{n}.address docs indicating none is valid value
- shared/util/linux: Add channel closed check before writing in ExecReaderToChannel
- lxd/metrics: OpenMetrics says to end with EOL
- lxd: Rename metrics to api_metrics
- lxd/daemon: Drop metrics from main struct
- lxd/metrics: Rework caching and locking
- lxd/metrics: Reduce cache to 8s to accomodate 10s intervals
- lxc/console: Don’t write twice to sendDisconnect
- doc/api-extensions: Remove trailing whitespaces
- lxd: Check serverName in clusterMemberJoinTokenDecode
- shared/api/cluster: s/Base64/base64/
- lxd/certificates: Fix typo in comment
- lxc/config_trust: Don’t remove extension from cert
- shared: Remove SplitExt
- lxd/certificates: Rework certificate name logic
- doc: download external images
- doc: use local image
- doc: add doc output to “make dist”
- shared/api: Add CertificateAddToken
- lxd/instance/drivers/driver/lxc: Reduce calls to VolatileSet to reduce DB transactions
- lxd/instance/drivers/load: Removes unused cluster arg from validDevices
- lxd/instance/drivers: instance.ValidDevices usage
- lxd/instance/instance/utils: ValidDevices definition and usage
- lxd/profiles/utils: instance.ValidDevices usage
- lxd/profiles: instance.ValidDevices usage
- lxd/project/permissions: Updates CheckClusterTargetRestriction to accept a project record
- lxd/project/permissions/test: project.CheckClusterTargetRestriction usage
- lxd/instances/post: project.CheckClusterTargetRestriction usage
- lxd/instance/post: project.CheckClusterTargetRestriction usage
- lxd/cluster/config: Add ImagesDefaultArchitecture function
- lxd/instances/post: Uses config.ImagesDefaultArchitecture
- shared/api: Add ToCertificateAddToken
- shared/api: Add Token to CertificatesPost
- api: Add certificate_token
- shared: Add CertificateTokenDecode
- lxd/db: Add OperationCertificateAddToken
- lxd: Add certificateTokenValid
- lxd: Support certificate tokens
- client: Add CreateCertificateToken
- lxc: Support client tokens in lxc config trust
- lxc: Support client tokens in lxc remote add
- lxc: Add lxc config trust list-tokens
- lxc: Add lxc config trust revoke-token
- lxc/cluster: Make member arg optional when adding member
- test/suites/remote: Test client tokens
- doc/authentication: Add tokens
- i18n: Update translation templates
- lxd/instance/instance/utils: Remove call to tx.ProjectExists in CreateInternal
- lxd/instances/post: Removes unnecessary pool check in instancesPost
- lxd/db/query/retry: Improve consistency of logging in Retry
- lxd/instance/instance/utils: Comment fix
- lxd/instance/instance/utils: Don’t call ValidDevices multiple times during instance create
- lxd/db/networks: Update network load functions to share transaction
- lxd/instance/drivers: Merge expandDevices with expandConfig
- lxd/instances/post: Renames targetProject to targetProjectName and p to targetProject
- lxd/db/storage/pools: Reworks getStoragePool to use a single transaction internally
- shared/api: Add ToClusterJoinToken
- lxc/cluster: Switch to ToClusterJoinToken
- lxc/cluster: Drop clusterJoinTokenOperationToAPI
- lxc: Translate all errors
- i18n: Update translation templates
- lxd/db: Uses api.URL to build used-by urls for storage pools.
- doc: fix path to image in the header
- lxd/cluster/gateway: Enable TCP user timeout and connection closing on failure in dqliteNetworkDial
- lxd/cluster/gateway: Improve logging in dqliteNetworkDial
- lxd/cluster/gateway: Standardise logging naming of dqliteProxy and dqliteNetworkDial
- lxc: Properly report alias add/update errors
- i18n: Update translation templates
- client: Update example to use instances
- client: Clearly mark container functions as deprecated
- lxd/cgroup: Add total_cache on V2
- lxc/console: Rework concurency model for vga
- lxd/instance/drivers/driver/qemu: Add check for lxd-agent running in getAgentClient
- lxd/instance/drivers/driver/qemu: Remove duplicated lxd-agent running status check in agentGetState
- lxd-agent: Updates startStatusNotifier to return a cancel function
- lxd-agent: c.startStatusNotifier usage
- lxd/instance/drivers/qmp/monitor: Add agentReadyMu for proper shared access to agentReady
- lxd/instance/drivers/driver/qemu: Removed unused agentClient var
- github: Update for current min Go version (1.16)
- lxd-migrate: Create interactive tool
- i18n: Update translations from weblate
- doc/rest-api: Refresh swagger YAML
- gomod: Update dependencies
Try it for yourself
This new LXD release is already available for you to try on our demo service.
Downloads
The release tarballs can be found on our download page.
Binary builds are also available for:
- Linux: snap install lxd
- MacOS: brew install lxc
- Windows: choco install lxc