Introduction
The Incus team is pleased to announce the release of Incus 0.6!
This second release of 2024 features a number of improvements, both large and small.
It comes with a completely new storage driver for cluster users, import/export support for storage buckets, a number of improvements for OVN users, support for new container kernel features, quite a few improvements to the migration tooling and more!
As usual, you can try it for yourself online: Linux Containers - Incus - Try it online
Enjoy!
New features
Clustered LVM storage driver
A new storage driver named lvmcluster has been added.
As the name implies, this is for clustered use of the LVM storage driver.
This relies on llvmlockd and a lock manager like sanlock to provide concurrent access to a shared storage device across a cluster.
This is aimed at anyone who wants to run an Incus cluster and use a shared storage device like a FiberChannel SAN, iSCSI export, NVMEoF/NVMEoTCP disk, …
With the ability of having the same storage available on all servers comes support for much faster live migrations, server evacuations and the ability to automatically recover should a server suddenly die.
Documentation: LVM - lvm - Incus documentation
Storage bucket backup and import
It is now possible to backup a full storage bucket and later re-import back into Incus.
stgraber@dakara:~$ incus storage bucket create default foo
Storage bucket foo created
Admin access key: HD2GWC1KX693MFVT3P30
Admin secret key: tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv
stgraber@dakara:~$ dd if=/dev/random of=out.img bs=4M count=10
10+0 records in
10+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.0975758 s, 430 MB/s
stgraber@dakara:~$ s3cmd --host=127.0.0.1:8445 --host-bucket=127.0.0.1:8445 --no-check-certificate --access_key=HD2GWC1KX693MFVT3P30 --secret_key=tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv put out.img s3://foo
upload: 'out.img' -> 's3://foo/out.img' [part 1 of 3, 15MB] [1 of 1]
15728640 of 15728640 100% in 0s 262.42 MB/s done
upload: 'out.img' -> 's3://foo/out.img' [part 2 of 3, 15MB] [1 of 1]
15728640 of 15728640 100% in 0s 241.54 MB/s done
upload: 'out.img' -> 's3://foo/out.img' [part 3 of 3, 10MB] [1 of 1]
10485760 of 10485760 100% in 0s 241.99 MB/s done
stgraber@dakara:~$ s3cmd --host=127.0.0.1:8445 --host-bucket=127.0.0.1:8445 --no-check-certificate --access_key=HD2GWC1KX693MFVT3P30 --secret_key=tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv ls s3://foo
2024-02-23 03:26 41943040 s3://foo/out.img
stgraber@dakara:~$ incus storage bucket export default foo
Backup exported successfully!
stgraber@dakara:~$ incus storage bucket delete default foo
Storage bucket foo deleted
stgraber@dakara:~$ incus storage bucket import default backup.tar.gz
stgraber@dakara:~$ s3cmd --host=127.0.0.1:8445 --host-bucket=127.0.0.1:8445 --no-check-certificate --access_key=HD2GWC1KX693MFVT3P30 --secret_key=tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv ls s3://foo
2024-02-23 03:27 41943040 s3://foo/out.img
API: Main API specification - Incus documentation
Listing image across all projects
Just as it’s possible to list instances across all projects using the --all-projects, it’s now possible to do the same but for images.
stgraber@dakara:~$ incus image list --all-projects
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| PROJECT | ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCHITECTURE | TYPE | SIZE | UPLOAD DATE |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| default | | 256f59a72af5 | no | Ubuntu jammy amd64 (20240222_07:42) | x86_64 | VIRTUAL-MACHINE | 267.19MiB | Feb 23, 2024 at 12:27am (UTC) |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| default | | 0941e441dbb9 | no | Alpine edge amd64 (20240222_13:00) | x86_64 | CONTAINER | 2.93MiB | Feb 23, 2024 at 12:27am (UTC) |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| default | | d5fc6024f0fa | no | Openwrt snapshot amd64 (20240222_11:57) | x86_64 | CONTAINER | 3.50MiB | Feb 23, 2024 at 2:55am (UTC) |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| demo | | f44a6b4e56f4 | no | Archlinux current amd64 (20240222_04:18) | x86_64 | CONTAINER | 188.86MiB | Feb 23, 2024 at 3:32am (UTC) |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
Over the API, this is done by passing ?all-projects=true.
binfmt_misc in unprivileged containers
Linux 6.7 added support for mounting of binfmt_misc inside of unprivileged containers.
Incus 0.6 will detect kernels that support this feature and when they do, it will no longer bind-mount binfmt_misc from the host system but instead allow it to be mounted from within the container.
stgraber@castiana:~$ incus launch images:ubuntu/22.04 foo
Launching foo
stgraber@castiana:~$ incus exec foo bash
root@foo:~# uname -a
Linux foo 6.7.4-zabbly+ #debian12 SMP PREEMPT_DYNAMIC Mon Feb 5 23:37:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
root@foo:~# mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/
root@foo:~# ls -lh /proc/sys/fs/binfmt_misc/
total 0
--w------- 1 root root 0 Feb 23 03:46 register
-rw-r--r-- 1 root root 0 Feb 23 03:46 status
The binfmt_misc filesystem is used to define custom binary formats for emulation.
It’s most notably used by qemu-user-static as a way to run binaries of other architectures.
Control over shared block storage volumes
A new security.shared volume storage option has been introduced to control whether a custom block volume should be allowed to be used by multiple instances.
It is now required to set that configuration key prior to adding a custom block volume to a profile or add it directly to multiple virtual machines.
stgraber@castiana:~$ incus config device add lvm01 shared disk pool=default source=shared-lvm io.bus=nvme
Device shared added to lvm01
stgraber@castiana:~$ incus config device add lvm02 shared disk pool=default source=shared-lvm io.bus=nvme
Error: Failed add validation for device "shared": Cannot add un-shared custom storage block volume to more than one instance
stgraber@castiana:~$ incus storage volume set default shared-lvm security.shared=true
stgraber@castiana:~$ incus config device add lvm02 shared disk pool=default source=shared-lvm io.bus=nvme
Device shared added to lvm02
stgraber@castiana:~$ incus config device add lvm03 shared disk pool=default source=shared-lvm io.bus=nvme
Device shared added to lvm03
stgraber@castiana:~$ incus start lvm01 lvm02 lvm03
OVN logical router name in network info
To make it easier to see what’s going on within OVN, incus network info now gives you the name of the logical router for a particular network.
root@abydos:~# incus network info default
Name: default
MAC address: 00:16:3e:38:dd:28
MTU: 1500
State: up
Type: broadcast
IP addresses:
inet 10.180.103.1/24 (link)
inet6 2602:fc62:a:1004::1/64 (link)
Network usage:
Bytes received: 0B
Bytes sent: 0B
Packets received: 0
Packets sent: 0
OVN:
Chassis: abydos
Logical router: incus-net6-lr
File ownership and permissions in image templates
Template files can now have a uid, gid and mode set on them.
This can be particularly useful if a template is meant to be an executable shell script.
stgraber@castiana:~$ incus config metadata show foo
architecture: amd64
creation_date: 1708588077
expiry_date: 1711180077
properties:
architecture: amd64
description: Ubuntu jammy amd64 (20240222_07:42)
name: ubuntu-jammy-amd64-default-20240222_07:42
os: ubuntu
release: jammy
serial: "20240222_07:42"
variant: default
templates:
/etc/hostname:
when:
- create
- copy
create_only: false
template: hostname.tpl
properties: {}
/etc/hosts:
when:
- create
- copy
create_only: false
template: hosts.tpl
properties: {}
/root/hello.sh:
when:
- start
create_only: false
template: hello.tpl
properties: {}
uid: "1000"
gid: "2000"
mode: "0755"
stgraber@castiana:~$ incus config template show foo hello.tpl
#!/bin/sh
echo "Hello world!"
stgraber@castiana:~$ incus start foo
stgraber@castiana:~$ incus exec foo bash
root@foo:~# ls -lh /root/hello.sh
-rwxr-xr-x 1 ubuntu 2000 30 Feb 23 04:07 /root/hello.sh
root@foo:~# /root/hello.sh
Hello world!
Documentation: Image format - Incus documentation
Encrypted EC client certificate keys
For those interacting with remote Incus servers, you may not know that it’s possible to protect your Incus private key with a password.
Up until now, this was only possible for RSA keys, but with Incus 0.6, we’re now adding support for EC keys too. That’s particularly relevant as EC keys have been the default for a while now.
Documentation: Remote API authentication - Incus documentation
Worth noting that if using this feature, you’ll likely also want to make use of the recently introduced “keepalive mode” as this will then significantly reduce the number of password prompts you’ll get while using Incus.
lxd-to-incus improvements
Lastly, lxd-to-incus saw a number of improvements:
- Support for Void Linux
- Detection of the
boot.debug_edk2configuration key - Handling of OVN SSL database connections
- Automatic clearing of the simplestreams cache during migration
Complete changelog
Here is a complete list of all changes in this release:
Full commit list
- incusd/instance/qemu/qmp: Don’t risk blocking QMP on eject
- github: Switch to minio deb and pin working version
- github: Use stable MicroCeph
- incusd/db: Fix error handling in CreateNetworkPeer
- incusd/apparmor: Allow more mounts in unprivileged containers
- Translated using Weblate (Portuguese (Brazil))
- incusd/instance/qemu/agent-loader: Handle legacy lxd-agent-loader
- incusd/device/disk: Use genisoimage when mkisofs can’t be found
- shared/api: Add example of Chassis in NetworkStateOVN
- doc/rest-api: Refresh swagger YAML
- api: network_state_ovn_lr
- shared/api: Add LogicalRouter field to NetworkStateOVN
- incusd/network/ovn: Expose LogicalRouter name
- incus: Add OVN logical router name
- i18n: Update translation templates
- doc/rest-api: Refresh swagger YAML
- scripts/bash: Don’t follow aliases
- Translated using Weblate (Japanese)
- test: Adapt test_database_no_disk_space() to newer libraft versions
- incus/launch: Fix --console
- api: image_template_permissions
- doc: Add uid/gid/mode to image template metadata
- shared/api: Add owner/permission fields to ImageMetadataTemplate
- doc/rest-api: Refresh swagger YAML
- incusd/instance/lxc: Add support for template ownership/mode
- incus-agent: Add support for template ownership/mode
- lxd-to-incus: Add comment
- lxd-to-incus: Add cache cleanup logic
- .github: Bump minimum Go version to 1.21
- Makefile: Bump minimum Go version to 1.21
- gomod: Update dependencies
- doc: Bump minimum Go version to 1.21
- Translated using Weblate (French)
- incusd/daemon: Fix log format
- incusd/sys: Sort kernel features
- incusd/sys: Add doc strings
- incusd/checkfeature: Add unprivileged binfmt detection
- incusd/sys: Add UnprivBinfmt
- incusd/daemon: Detect unprivileged binfmt_misc
- incusd/api_1.0: Add unpriv_binfmt kernel feature
- incusd/instance/lxc: Support unpriv binfmt_misc
- incusd/apparmor: Support unpriv binfmt_misc
- Translated using Weblate (Italian)
- completion: support returning non-incus remotes
- incus/image: add dynamic command line completions
- shared/cliconfig: Support SSH encrypted keys
- tests: Add crypto/ssh
- doc/remotes: Add mention of keepalive
- doc: Add Ansible to wordlist
- doc/authentication: Hints on encrypting client key
- api: images_all_projects
- lxd-to-incus: Indicate what existing configuration was found
- shared/api: Add project property to Image
- client: Add GetImagesAllProjects
- incusd/images: Add support for all_projects
- doc/rest-api: Refresh swagger YAML
- incus/image: Add --all-projects flag to list
- i18n: Update translation templates
- shared/cliconfig: Fix static analysis
- gomod: Update dependencies
- Makefile: Pin rpc2 version due to OVN issue
- incusd/instance/qemu: Re-shuffle agent NIC handling
- incusd/instance/qemu/agent-loader: Handle SELinux
- incus/config: add completions
- incus-agent: Load virtio_net before configuring NICs
- incus: Run gofmt
- doc/storage/lvm: Cleanup tables
- incusd/patches: Move lvm.vg.force_reuse to be server-specific
- incusd/db: Mark lvm.vg.force_reuse as server-specific
- doc/faq: Add mention of the -mtu network interfaces
- incusd/device: Bump base VM filesystem volume to 500MiB
- incusd/storage: Create rootfs dir on empty instances
- incusd/storage/quota: Don’t fail on missing project
- incusd/instances: Use correct project on cross-project copy
- incusd/patches: Fix bad SQL query
- incusd/instances: Don’t bypass instance limit check
- Added translation using Weblate (Portuguese)
- incus-agent: Re-order imports
- api: Add storage_bucket_backup extension
- shared/api: Add storage bucket backup
- incusd/db: Add storage bucket backup functions
- incusd/db/operation: Add storage volume backup types
- incusd/lifecycle: Add storage bucket backup events
- incusd/project: Add StorageBucket function
- incusd/storage/s3: Add transfer manager
- incusd: Add storage bucket backup
- client: Add storage bucket backup
- incus: Add storage bucket import/export
- doc/rest-api: Refresh swagger YAML
- i18n: Update translation templates
- test: Add storage bucket backup
- client: Remove ceph-specific logic
- incusd/storage/s3: Fix typo
- incus: Fix import shadowing
- incus: Fix comments on exported functions
- client: Fix comments on exported functions
- incusd: Fix import shadowing
- incusd: Fix comments on exported functions
- incusd/cluster: Make remote storage volume logic generic
- incusd/db: Don’t hardcode remote storage drivers
- incusd/storage_volumes: Don’t hardcode ceph
- incusd/instances: Update ceph-specific comment
- incusd/cluster: Update ceph-specific comment
- incusd/instances: Generalize Ceph logic
- lxd-to-incus: Support SSL authentication in OVN
- lxd-to-incus: Don’t fail on missing OVN bridge mapping
- lxd-to-incus: Ignore OVN chassis external_ids
- lxd-to-incus: Add boot.debug_edk2 to deprecated keys
- lxd-to-incus: Support Void Linux
- cmd/incusd: Disable the architecture check on incus cp/mv
- incusd/instance: Disable the architecture check on incus cp/mv
- api: storage_lvm_cluster
- incus/admin/init: Add LVM cluster logic
- i18n: Update translation templates
- lxd-to-incus: Add LVM cluster
- doc: Add LVM cluster
- shared/api: Add cephobject
- shared/api: Add lvmcluster
- doc/rest-api: Refresh swagger YAML
- incusd/storage/lvm: Add clustered LVM
- incusd/storage/lvm: Fix import shadowing
- incusd/storage/lvm: Tweak locking in cluster
- internal/server/device/config: Don’t include empty values
- internal/server/device/config: Fix return values of Update function
- api: Add shared_custom_block_volumes API extension
- doc/reference: Add
security.sharedconfig key - instance/server/device: Validate shared block devices
- internal/server/storage: Handle
security.sharedupdate - internal/server/storage: Allow security.shared key for custom block volumes
- internal/server/storage/drivers: Handle security.shared in fillVolumeConfig
- incusd/instance/qemu: Cap hotplug CPU slots to 64
- incusd/storage/lvm: Make gofmt happy
- shared/subprocess: Improve error handling
- incusd/instance/qemu: Improve error handling
- incusd/network/ovs: Wait for bridge interface to appear
- incusd/storage/zfs: Fix refresh of VM volumes
- internal/instance: Fix volatile key definitions
- doc: Update configs
- incusd/network/ovn: Extend validateExternalSubnet to allow uplink subnets
- incusd/network/ovn: Add static routes to load-balancer and forwards
- gomod: Update dependencies
Documentation
The Incus documentation can be found at:
Packages
There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.
Installing the Incus server on Linux
Incus is available for most common Linux distributions. You’ll find detailed installation instructions in our documentation.
Homebrew package for the Incus client
The client tool is available through HomeBrew for both Linux and MacOS.
Chocolatey package for the Incus client
The client tool is available through Chocolatey for Windows users.
Winget package for the Incus client
The client tool is also available through Winget for Windows users.
https://winstall.app/apps/LinuxContainers.Incus
Support
At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.
Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: Zabbly - Incus services
Bugs can be reported at: Issues · lxc/incus · GitHub
