Incus 6.10 has been released

News
,
1

Introduction

The Incus team is pleased to announce the release of Incus 6.10!

This release brings in an easier way to run Incus on a valid HTTPS certificate, a new way to send through provisioning data to VMs, a very welcome API enhancement and much more!

image

As usual, you can try it for yourself online: Linux Containers - Incus - Try it online

Enjoy!

New features

ACME DNS-01 validation

New configuration keys have been added to allow for the DNS-01 challenge through ACME (most commonly Let’s Encrypt).

This may provide an easier path at getting a valid TLS certificate on Incus servers.

Example configuration:

acme.agree_tos: true
acme.domain: foo.example.net
acme.email: admin@foo.example.net
acme.challenge: DNS-01
acme.provider: cloudflare
acme.provider.environment: |-
  CLOUDFLARE_EMAIL=admin@foo.example.net
  CLOUDFLARE_API_KEY=XYZ

Documentation: Server configuration - Incus documentation

API wide filtering support

Incus already supported server-side filtering for instances, images and storage volumes.
The same logic has now been expanded to all remaining API collections.

All of them support the filter parameter with the same OData filters.
We expect to see the Incus command line tool making use of this over the next few releases.

Documentation: REST API - Incus documentation

Support for SMBIOS11 provisioning in VMs

On systems with DMI tables (x86_64 or aarch64), it is now possible to seed data into the SMBIOS Type 11 table. This is a way for the hypervisor to provide data to the system without requiring a separate communication channel (network, serial, …).

This is most notably supported by systemd where it can be used to pass in some provisioning data: smbios-type-11

Example:

stgraber@dakara:~$ incus launch images:debian/12 d12 --vm -c smbios11.io.systemd.credential:foo=bar
Launching d12
stgraber@dakara:~$ incus exec d12 -- systemd-creds --system cat foo
bar

IOMMU support in VMs

Incus VMs now automatically get IOMMU support which will help with security and device passthrough.

This was introduced as default behavior in Incus 6.10.0, but following serveral reports of guest kernel and host platform combinations that led to broken PCIe in the guest, it has been moved to opt-in starting with Incus 6.10.1 using the security.iommu configuration option to control it.

root@u1:~# dmesg | grep -i iommu
[    0.406474] iommu: Default domain type: Translated
[    0.406481] iommu: DMA domain TLB invalidation policy: strict mode
[    0.534812] virtio_iommu virtio0: input address: 48 bits
[    0.534820] virtio_iommu virtio0: page mask: 0xfffffffffffff000
[    0.637345] pcieport 0000:00:02.0: Adding to iommu group 0
[    0.642755] pcieport 0000:00:02.1: Adding to iommu group 1
[    0.649614] pcieport 0000:00:02.2: Adding to iommu group 2
[    0.656700] pcieport 0000:00:02.3: Adding to iommu group 3
[    0.663517] pcieport 0000:00:02.4: Adding to iommu group 4
[    0.669419] pcieport 0000:00:02.5: Adding to iommu group 5
[    0.675831] pcieport 0000:00:02.6: Adding to iommu group 6
[    0.682967] pcieport 0000:00:02.7: Adding to iommu group 7
[    0.688303] pcieport 0000:00:03.0: Adding to iommu group 8
[    0.695646] pcieport 0000:00:03.1: Adding to iommu group 9
[    0.700994] pcieport 0000:00:03.2: Adding to iommu group 10
[    0.709613] pcieport 0000:00:03.3: Adding to iommu group 11
[    0.715130] pcieport 0000:00:03.4: Adding to iommu group 12
[    0.722208] virtio-pci 0000:01:00.0: Adding to iommu group 13
[    0.728419] virtio-pci 0000:01:00.1: Adding to iommu group 13
[    0.732362] virtio-pci 0000:01:00.2: Adding to iommu group 13
[    0.737953] virtio-pci 0000:01:00.3: Adding to iommu group 13
[    0.746652] virtio-pci 0000:01:00.4: Adding to iommu group 13
[    0.752491] virtio-pci 0000:01:00.5: Adding to iommu group 13
[    0.781052] virtio-pci 0000:02:00.0: Adding to iommu group 14
[    0.791255] virtio-pci 0000:03:00.0: Adding to iommu group 15
[    0.799383] virtio-pci 0000:03:00.1: Adding to iommu group 15
[    0.809692] virtio-pci 0000:04:00.0: Adding to iommu group 16
[    0.820005] virtio-pci 0000:05:00.0: Adding to iommu group 17
[    1.399721] xhci_hcd 0000:01:00.6: Adding to iommu group 13

VRF support for routed NICs

Systems using multiple VRFs (Virtual Routing and Forwarding) can now have specific routed NICs land in the VRF of their choice.

This is done through a new vrf property on routed type nic devices.

Documentation: Type: nic - Incus documentation

Creating profiles in a project through preseed

The Incus server preseed logic has been expanded to allow for profiles to be created in specific projects.

This is done through a new project key on the relevant profile entry.
The project then needs to also be defined within the preseed.

Documentation: How to initialize Incus - Incus documentation

LZ4 support for backups and images

lz4 is now a fully supported compression format within Incus.
Provided the matching tool is installed on the system, it’s now possible to use LZ4 for both instance, volume and bucket backups (import/export) as well as for images.

Complete changelog

Here is a complete list of all changes in this release:

Full commit list
  • incusd/instance/drivers/qmp: Handle missing log directory
  • cmd/incus-user: keep track of socket path used to connect to the server
  • cmd/incus-user: unify logging, support --verbose and --debug
  • api: init_preseed_profile_project
  • shared/api: Add project support to profiles in preseed init
  • client: Add project support to profiles in preseed init
  • doc/rest-api: Refresh swagger YAML
  • incus/admin/init: use api.InitProfileProjectPost
  • incusd/network/ovn: Fix bad route check
  • incus/file/pull: Ensure we have a leading / in all paths
  • github: Fix shellcheck
  • doc/installing: mention incus group on NixOS
  • incus/file/pull: Read files in chunks
  • incus/file/pull: Actually make read buffer 1MiB
  • incusd/network/bridge: Fix deletion of tunnels and dummy devices
  • incusd/device/disk: Allow virtiofsd on non-x86
  • Translated using Weblate (Chinese (Simplified Han script))
  • Translated using Weblate (Chinese (Simplified Han script))
  • Translated using Weblate (Chinese (Simplified Han script))
  • Translated using Weblate (Chinese (Simplified Han script))
  • Translated using Weblate (Chinese (Simplified Han script))
  • Translated using Weblate (Chinese (Simplified Han script))
  • incusd/instance/drivers/qemu: Add IOMMU device
  • incus/file: Remove unused function
  • i18n: Update translation templates
  • incus/network/info (ovn): Fix object not found.
  • Translated using Weblate (Chinese (Simplified Han script))
  • incusd/instance/drivers: Improve NUMA balancing
  • incus/file: Move from path to filepath
  • github: Add linux-modules-extra (for vrf)
  • shared/archive: Added LZ4 support
  • api: instance_nic_routed_host_address
  • incus/server/ip: Add Master parameter for veth and tuntap
  • incus/server/ip: Add VRF parameter for IP Route
  • incus/server/device: Add vrf parameter for routed NIC devices
  • tests: Add test for routed NIC with VRF
  • doc: Add description for routed-nic VRF parameter
  • Translated using Weblate (Chinese (Simplified Han script))
  • Translated using Weblate (Chinese (Simplified Han script))
  • generate-database: Move into cmd package
  • generate-config: Move to cmd package
  • incus/storage/drivers: Fix issue with lvmcluster storage pool creation
  • generate-database: Rename from incus-generate
  • generate-config: Rename from incus-doc
  • github: Workaround permission problems
  • internal/instance: Move user config key doc string
  • internal/instance: Add environment config keys docstring
  • incusd/internal/server/instance: Add qemuEscapeCmdline
  • incusd/internal/server/instance: Use qemuEscapeCmdline
  • api: instance_smbios
  • internal/instance: Add smbios config keys
  • incusd/instance/drivers/qemu: Implement smbios config keys
  • doc: Update configs
  • incus: Fix instance copy error when using ‘–refresh’ flag
  • doc/network_load_balancers: Fix backend add instructions
  • doc: Add frontend to wordlist
  • incusd/main_nsexec: Fix change_namespaces fallback to handle multiple namespaces
  • incusd/instance/utils: Only check uid/gid for containers
  • generate-database: Use types.Package
  • generate-database: Remove support for separate database package
  • generate-database: Separate Go package loading
  • incusd/instance/drivers: Check if disk is remote when migrating with an extra disk
  • incusd/instance/edk2: Look for bios.bin in /usr/share/seabios
  • generate-database: Add unified generate command
  • incusd/db: Use generate command
  • generate-database: Remove obsolete commands stmt and method
  • generate-database: Update README.md
  • generate-database: Reduce to single go:generate per package
  • incusd/db: Use Single generate per package
  • golangci: Update for new generate-database syntax
  • internal/cmd/table: Improve errors on modifiers
  • incusd: Add additional validation when joining a new cluster member
  • incusd: Upgrade flosch/pongo2 to v6
  • doc: Bump minimum kernel to 5.15
  • Makefile: Bump minimal Go to 1.23
  • doc: Bump minimum Go to 1.23
  • gomod: Update dependencies
  • github: Re-order code tests tasks
  • incusd/resources: Prevent concurrent runs and cache data for 10s
  • incusd/backup: Implement compatibility with old backups
  • incusd/backup: Detect incompatible backup files
  • fix: Don’t attempt to download signatures for oci
  • incusd: Ensure directories have 755 permissions in ‘incus file push -p’ command
  • tests: Fix test for verifying directory permissions
  • devcontainer: Update Go to 1.23
  • generate-database: Make “Code generated” comment Go conformant
  • db/cluster: Update code generated by generate-database
  • generate-database: Add internal tool disclaimer
  • generate-config: Formatting README.md
  • generate-config: Add internal tool disclaimer
  • incusd/storage/drivers: Truncate the block file during custom volume migration
  • incusd/instance/qemu: Don’t attempt to hotplug over the virtual IOMMU
  • incusd/device/disk: Remove diskAddRootUserNSEntry
  • incusd/device/disk: Use virtiofsd uid/gid handling
  • generate-database: Add error mapping
  • incusd/db: Use error mapping
  • incusd/db: Update generated code
  • generate-database: Make self-sufficient
  • incusd/db: Update generated code
  • generate-database: Improve case handling
  • incus/list: Remove unused arguments and parameters
  • incus/top: Fix handling of all-projects
  • incusd/storage/drivers: Refactor ceph discovery
  • incusd/instance/drivers: Let QEMU handle Ceph itself
  • incus/file: Simplify sftpCreateFile
  • incus/file: Change recursiveFilePush to use SFTP
  • incus/file: Change recursiveMkdir to use SFTP
  • incus/file: Change recursivePullFile to use SFTP
  • incus/file: Re-use SFTP as much as possible
  • internal/filter: Handle multiple in-line structs
  • api: api_filtering_extended
  • incusd: Add filtering to /1.0/networks
  • incusd: Add filtering to /1.0/networks/{networkName}/forwards
  • incusd: Add filtering to /1.0/networks/{networkName}/load-balancers
  • incusd: Add filtering to /1.0/networks/{networkName}/peers
  • incusd: Add filtering to /1.0/network-acls
  • incusd: Add filtering to /1.0/network-integrations
  • incusd: Add filtering to /1.0/network-zones
  • incusd: Add filtering to /1.0/network-zones/{zone}/records
  • incusd: Add filtering to /1.0/storage-pools
  • incusd: Add filtering to /1.0/storage-pools/{poolName}/buckets
  • incusd: Add filtering to /1.0/certificates
  • incusd: Add filtering to /1.0/profiles
  • incusd: Add filtering to /1.0/projects
  • doc/rest-api: Refresh swagger YAML
  • tests: Test more API filters
  • doc: Remove mention of limitations on filtering
  • gomod: Update dependencies
  • api: acme_dns01
  • incusd/cluster/config: Add extra ACME config keys
  • incusd/acme: Add DNS-01 support
  • gomod: Update dependencies
  • doc: Add resolvers to wordlist
  • doc: Update configs

Documentation

The Incus documentation can be found at:

Packages

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Installing the Incus server on Linux

Incus is available for most common Linux distributions. You’ll find detailed installation instructions in our documentation.

Homebrew package for the Incus client

The client tool is available through HomeBrew for both Linux and MacOS.

Chocolatey package for the Incus client

The client tool is available through Chocolatey for Windows users.

Winget package for the Incus client

The client tool is also available through Winget for Windows users.

https://winstall.app/apps/LinuxContainers.Incus

Support

Monthly feature releases are only supported up until the next release comes out. Users needing a longer support length and less frequent changes should consider using Incus 6.0 LTS instead.

Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: Zabbly - Incus services
Bugs can be reported at: GitHub · Where software is built

3 Likes
3
1 Like
4

Note that a regression has been found in Incus 6.10 affecting Ceph users.

The new Ceph logic relies on QEMU being able to read the Ceph keyring directly.
That’s typically not possible in properly configured environments as permissions on Ceph keyring are (and should be) pretty restricted. Incus’ QEMU processes run as an unprivileged user and therefore aren’t allowed access.

A fix for this is currently going through testing and will be cherry-picked into the Zabbly packages as soon as it’s been merged.

Until then, starting instances on such systems may fail with “Operation not permitted”.
A workaround is to either change the ownership of the keyring to the incus user and/or group or to temporarily set permissions to be world readable (644).

In either case, do keep in mind that you may be exposing access to your RBD keyring to users that would otherwise not have access so you may want to make sure that only trusted users can have filessytem access to the system while the workaround is in place.

UPDATE: The fix has been merged and updated packages should start appearing in around 3h (2025-03-01T12:00:00Z)

1 Like
5

The ceph bug sounds problematic, glad you caught it and found a patch quickly. As a packager, it would be very helpful if fixes like this could end up in a release. Is there consideration for cutting patch releases especially for serious bugs?

The patches are not terribly difficult to apply, but tracking releases is much easier and more discoverable. If I hadn’t delayed a few days and scrolled down to read this entire thread, I likely would have missed this fix and shipped to my users without it.

1 Like
6

With incus, everything’s a rolling release.

For example, although I’m on 6.0 LTS, and there are tagged versions in git like v6.0.0 … v6.0.3, what I actually get in the apt packages are intermediate versions from the stable-6.0 branch, released every couple of weeks, for example:

1:6.0.3-ubuntu22.04-202412201834
1:6.0.3-ubuntu22.04-202501311434

Therefore, I think it’s likely that you can expect an update to arrive soon for 6.10, and it might not even be called 6.10.1.

(Personally I’m not a fan of rolling releases; I’d rather all the changes were rolled up into versioned releases, so when I install v6.0.3, say, I know exactly what I’m getting. But that’s just how it is.)

1 Like
7

I don’t follow the zabbly packages close enough, so I’m not sure what’s changing in between those point releases. If they’re important fixes, they’re being missed by us in nixpkgs since we target the released versions. In general, we’re not pulling in unreleased fixes unless they’re actively fixing incompatibilities we have with newer software. e.g. if nixpkgs updates qemu we may need to backport some unreleased patches.

So for our users, there is no “rolling release”, there’s the published releases with the occasional fix added. That LTS is potentially getting patched more frequently in the zabbly packages gives me even more concern that we’re missing fixes. Unfortunately, I don’t have the time to follow incus development closely enough to know which PRs are important enough to ship to our users, or even ones that the zabbly packages are bringing in.

8

Because of the Ceph and IOMMU issues, we’re looking at a 6.10.1.
We usually don’t bother as that’s pretty high effort considering we do a release a month anyway, but given the importance of those two issues, we’ll make an exception this time.

3 Likes
9

Yeah, I appreciate that. I’ve found a number of NixOS using the non-LTS, even though it’s not our default, and even mildly impacting issues can be frustrating for a month.

10

Incus 6.10.1 announcement

Incus 6.10.1 has been released as a bugfix release on top of Incus 6.10 fixing a few regressions that were reported soon after release.

This corrects four regressions:

  • Addition of IOMMU causing boot issues with some Linux kernels and host platforms.
    To limit the impact, IOMMU is now opt-in through security.iommu, default behavior is the same as in Incus 6.9.
  • Fixed a Ceph regression when the unprivileged Incus user wasn’t able to directly read the Ceph keyring
  • Fixed a Ceph regression when using a non-default cluster name
  • Fixed a Ceph regression when using a restricted (rbd-only) client keyring for Incus

Full changelog

 - incusd/network/bridge: Fix deletion of tunnels and dummy devices
 - incusd/device/disk: Include cluster name and config path in RBD string
 - incusd/instance/qemu: Provide the RBD keyring to QEMU
 - Translated using Weblate (Chinese (Simplified Han script))
 - api: security_iommu
 - internal/instance: Add security.iommu
 - doc: Update configs
 - incusd/instance/qemu: Simplify Windows detection
 - incusd/instance/qemu: Implement security.iommu
 - incusd/instance/qemu: Add IOMMU for Windows
 - incusd/instance/qemu: Configure irqchip for split mode with IOMMU
 - doc: Add passthrough and IOMMU to wordlist
 - incusd/storage/ceph: Re-introduce keyring parsing
3 Likes
11

6.10 release ACME DNS-01 validation feature introduced ~ 100MB binary to incusd:

cd cmd/incusd
go build -tags libsqlite3
du -hs incusd #=>
178M

Is it worth? maybe we can reimplement the feature via lego binary.

1 Like
12

(post deleted by author)

13

Yeah, lego is pretty intense as far as what it pulls in.
Speaking of on-disk footprint, having the support directly in incusd leads to a smaller combined foorprint (saves about 20MB).

But the memory footprint is a bit more of a problem, so I think it makes sense to drive it externally.

14