Incus 0.1 has been released

Introduction

The Linux Containers team is very excited to announce the initial release of Incus!

Incus is a community fork of Canonical LXD, created by @cyphar and now part of the Linux Containers project.

This initial release is roughly equivalent to LXD 5.18 but with a number of breaking changes on top of the obvious rename.

You can try it for yourself online: Linux Containers - Incus - Try it online

image

Changes

With this initial release of Incus, we took the opportunity to remove a lot of unused or problematic features from LXD. Most of those changes are things we would have liked to do in LXD but couldn’t due to having strong guarantees around backward compatibility.

Incus will be similarly strict with backward compatibility in the future, but as this is the first release of the fork, it was our one big opportunity to change things.

That said, the API and CLI are still extremely close to what LXD has, making it trivial if not completely seamless to port from LXD to Incus.

API

Removal of /1.0/containers and /1.0/virtual-machines

LXD started as a container-only project and therefore its REST API used /1.0/containers.

When virtual-machines were then introduced, a new /1.0/instances endpoint took over for all operations across both containers and virtual-machines, but /1.0/containers was kept around for legacy clients. On top of that, a convenience /1.0/virtual-machines endpoint was also added, though never used.

With Incus, those two legacy endpoints are now removed and the only supported way to interact with instances is through /1.0/instances.

Replacement of /dev/lxd with /dev/incus

As part of the renaming of everything from LXD to Incus, the guest API was also renamed from /dev/lxd to /dev/incus.

Convenience symlinks are currently provided so existing workloads will keep working after a migration from LXD.

Type change for the server configuration

The LXD server configuration has always been a bit oddly typed as map[string]any rather than our usual map[string]string. The reason for that one inconsistency was because of core.trust_password, the legacy authentication method used by LXD in the early days.

That’s because LXD never stored the password in clear text but instead would hash it and store the hash instead. This made it impossible to return the full configuration to the user. Instead when a password would be set, a true boolean would be returned rather than a string.

Now as mentioned, core.trust_password is a legacy way to handle authentication and should be replaced by either token based authentication, TLS certificate trust or external authentication.

Incus has removed support for core.trust_password completely (see below) and so all server config keys can now fit in a standard map[string]string.

Removal of legacy Container functions from the Go API

Following the above change of removing /1.0/containers and /1.0/virtual-machines, the matching functions in the Go client package are also all gone now.

Instead the Instance equivalents of those Container functions should be used.
(e.g. CreateInstance instead of CreateContainer)

CLI

Introduction of incus snapshot sub-command

A long term annoyance and inconsistency in the LXD CLI has been around snapshot management. While most other operations get their own sub-command (file, config, …), snapshots were kept at the top-level with snapshot and restore.

This was then forcing us to handle things like renaming or deleting snapshots through the lxc rename and lxc delete functions. Instead having an lxc snapshot sub-command with lxc snapshot create would have made a lot more sense but couldn’t be done without breaking all scripts using lxc snapshot.

Now is time to fix this, so incus snapshot is now a sub-command, featuring:

  • incus snapshot create
  • incus snapshot delete
  • incus snapshot list
  • incus snapshot rename
  • incus snapshot restore

Handling of incus config trust add and incus cluster add

With Incus doing away with password based authentication and focusing a lot more on tokens, it was important to make the experience of issuing tokens be easy and consistent.

As a result, both incus config trust add and incus cluster add now simply take a name as argument and return a valid token.

The certificate handling aspect of incus config trust add has now been moved to incus config trust add-certificate instead.

Introduction of incus admin sub-command

Another long time annoyance in the LXD world was the fact that both lxc and lxd had to be used to set up a server. With incus, the incusd binary can be safely hidden away as nobody should ever have to directly interact with it.

Instead, we now have the incus admin sub-command:

  • incus admin cluster
  • incus admin init
  • incus admin recover
  • incus admin shutdown
  • incus admin waitready

Dependencies

Transition to Cowsql

Shortly after @cyphar forked LXD as Incus, @freeekanayaka, the original author of Dqlite, also decided to fork Dqlite as Cowsql. Given that @freeekanayaka is also a maintainer on Incus, it made sense to port incus over to Cowsql.

Similarly to Incus itself, Cowsql is a community fork of Canonical Dqlite, largely compatible with Dqlite, allowing Incus to easily import existing data during the migration stage.

Focus on Cowsql so far has been on dramatically improving performance testing and making a number of changes across the Raft and Cowsql layers to make it as performant as possible for Incus’ usage pattern.

Minimum Go version

For those building Incus from source, the minimum Go version required is now Go 1.20.
In general, we’ll attempt to keep supporting building on the current and previous Go version.

Feature removal

A number of LXD features have been removed from Incus.
Those primarily focus around features that were added because of ecosystem reasons and/or depend on now deprecated or poorly maintained software.

Removal of Ubuntu Fan bridges

The Ubuntu Fan bridges depend on custom Ubuntu-only kernel patches and user-space changes to iproute2. While this feature is still maintained in the Ubuntu kernel, it can’t be found in the mainline kernel nor in any other distro’s kernel.

While convenient in some environments, Ubuntu Fan bridges are dwarfed by what’s possible through Incus’ integration with OVN.

The following network config keys have therefore been removed:

  • bridge.mode
  • fan.overlay_subnet
  • fan.underlay_subnet
  • fan.type

Removal of Ubuntu shiftfs

Another feature that’s unique to the Ubuntu kernel, shiftfs was an initial attempt at flexible uid/gid shifting at the kernel level.

While LXD supported shiftfs for years, it was never enabled by default due to a variety of kernel issues. Instead a cleaner in-kernel solution to this problem was developed, VFS idmap shifting. That’s now available in recent Linux kernels and automatically used by Incus when present.

Removal of Canonical Candid authentication

Back in the days, Canonical developed its own authentication system based on the use of Macaroons. The central authentication server for this was Candid.

LXD got Candid support, allowing it to authenticate users through a variety of providers through it.

These days, Candid is mostly unmaintained and the focus has been on moving towards industry standards, namely Open ID Connect.

Therefore, the following server configuration keys have been removed:

  • candid.api.key
  • candid.api.url
  • candid.domains
  • candid.expiry

Incus already has OpenID Connect support for external authentication and there are widely available OIDC servers that match and often exceed what Candid supported.

Removal of Canonical RBAC authorization

Canonical RBAC was a proprietary Role Based Access Control solution built on top of Macaroons and Candid.

It was only ever supported by MAAS and LXD and has seen extremely little adoption, mostly due to it being proprietary and generally difficult to get access to.

It’s mostly unmaintained and the focus these days is to transition to something more industry standard, namely OpenFGA.

Therefore the following server configuration keys have been removed:

  • rbac.agent.url
  • rbac.agent.username
  • rbac.agent.private_key
  • rbac.agent.public_key
  • rbac.api.expiry
  • rbac.api.key
  • rbac.api.url
  • rbac.expiry

Incus currently doesn’t have OpenFGA support, so any existing user of Canonical RBAC should stick with LXD until such time as an OIDC + OpenFGA alternative is available.

Removal of Canonical MAAS integration

LXD supports integrating with the MAAS API to use MAAS as some kind of IPAM (IP Address Management) solution.

This allows mapping specific LXD networks to MAAS subnet and then having MAAS create records for each instance on that network, assigning it a static address and DNS record.

Unfortunately, this integration has seen very little adoption and MAAS itself has a number of constraints that makes this integration problematic:

  • MAC addresses in MAAS are globally unique whereas in LXD they have to be unique among running instances on the same network. This different makes some instance copy operations impossible.
  • Instance names are expected to be globally unique in MAAS, even if attached to completely different DNS zones. This effectively makes it impossible to use LXD projects as instances can now easily conflict.

Between those issues and the lack of adoption of this feature, we’ve made the decision to remove MAAS support from Incus, the following configuration keys have therefore been removed:

  • maas.api.key
  • maas.api.url
  • maas.subnet.ipv4
  • maas.subnet.ipv6

Removal of the concept of trust password

Back in the early LXD days, the only way to connect to a remote server was to use a trust password, basically a fixed secret that would then allow a client to add its TLS certificate to the server’s trust store.

This effectively allowed anyone who knew or could find the trust password to add their client into the trust store after which the trust store was no longer required for further operations.

The trust password therefore had to be kept very safely, or unset immediately after adding a new client. This was unfortunately rarely the case, opening up the door for a brute force attack on the trust password and ultimately an attacker gaining full control over LXD and the server it runs on.

To address this, support for using one-time trust tokens has been added a little while back and all documentation updated to discourage users from using trust passwords.

In Incus, we went one step further and completely removed support for trust passwords. Adding new clients or new servers into a cluster should now be done through the use of one-time tokens or by using external authentication through OIDC.

Therefore, the following server configuration key has been removed:

  • core.trust_password

Other

Change in DMI information

Inside of Incus virtual machines, the vendor is now set to Linux Containers and the product is set to Incus.

Change in virtio-serial marker

The virtio-serial device used for limited communication with Incus prior to establishing full agent access through vsock is now org.linuxcontainers.incus.

Migration from LXD

Incus comes with a convenient lxd-to-incus tool which can be used to transition a system from LXD to Incus.

The main restrictions at this stage are:

  • Minimum LXD version of 4.0
  • Maximum LXD version of 5.18
  • No support for migrating clusters at this stage (this is being worked on)

Documentation

The Incus documentation can be found at:

Packages

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Zabbly packages for Debian and Ubuntu

Zabbly provides both daily and stable builds of Incus to Debian and Ubuntu users:

Homebrew package for the Incus client

The client tool is available through HomeBrew for both Linux and MacOS.

Chocolatey package for the Incus client

The client tool will soon be available through Chocolatey for Windows users.
Until then, binaries can be found here: Release Incus 0.1 · lxc/incus · GitHub

Support

At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.

Community support is provided at: https://discuss.linuxcontainers.org
Bugs can be reported at: Issues · lxc/incus · GitHub

23 Likes

Bravo! Already installed it, and it’s terrific! Keep up the good work.

2 Likes

Incus is now published on HomeBrew!

3 Likes

Is there ‘lxd-to-incus’ specific documentation I can take a look at ?

Not currently, the tool doesn’t really offer any flexibility though at this time.

What are you wondering about?

Simple example would be enough :slight_smile:

Let’s say I have LXD server at 1.2.3.4 IP and I want to migrate all containers from there to local Incus.

Is this possible? Or maybe I misunderstood how would this tool work…

lxd-to-incus does an in-place migration from LXD to Incus.

So you’d SSH to your server, add the Incus package repository, install the incus package and then run lxd-to-incus, the tool will make sure there’s no problematic configuration on the system and then ask you if you want to proceed with the migration.

At that point all instances will be stopped on LXD, the data will all be moved to Incus and then the instances will be started back, running under Incus at that point. Once the tool has confirmed that Incus is up and running, it will offer to remove the LXD package from the system.

3 Likes

I just published a video overview of our first release!

3 Likes

For testing, can LXD and Incus coexist on the same system? Is there compatibility to LXDWare LXD Dashboard by Matthew Penning wince the LXD GUI doesn’t support managing multiple nodes from the same GUI interface?

I don’t know about LXDWare but yes, Incus and LXD can be co-installed on the same system.

Thanks for taking your time in creating/posting this video. Helps a ton! Thanks again.

Couple of questions though:

  • Are there any extra steps / pre-reqs that one needs to do on the Debian host (shown in your demo) to have nested VMs running, if I have to create the same demo that you showcased (for my own learning purpose) ?

  • Any thoughts on supporting armbian, dietpi (based on armbian) that are debian based ? Those two run on most of the popular arm based embedded SBCs out there and perhaps can open up market for incus in that space. Just my humble opinion. anyways, thought I’ll ask :slight_smile:

Nope, no extra step here. Though if the VM you’re using isn’t running on Incus or LXD, there may be extra steps to enabled nested virtualization in the host’s virtualization solution.

Should work just fine if they have aarch64/arm64 builds. I provided packages for both amd64 and arm64. 32bit Arm isn’t all that interesting these days.

Wanted to confirm that the installation of incus 0.1 on radxa’s rockpi4c arm64 sbc board running dietpi was successful :slight_smile: I followed the instructions at GitHub - zabbly/incus: Incus package repository

-----
root@rockpi4c:~# uname -a
Linux rockpi4c 6.1.50-current-rockchip64 #3 SMP PREEMPT Wed Aug 30 14:11:13 UTC 2023 aarch64 GNU/Linux
root@rockpi4c:~#
root@rockpi4c:~# cat /etc/*release
PRETTY_NAME=“Armbian 23.8.3 bookworm”
NAME=“Debian GNU/Linux”
VERSION_ID=“12”
VERSION=“12 (bookworm)”
VERSION_CODENAME=bookworm
ID=debian
HOME_URL=“https://www.armbian.com
SUPPORT_URL=“https://forum.armbian.com
BUG_REPORT_URL=“Bug reporting – Armbian
ARMBIAN_PRETTY_NAME=“Armbian 23.8.3 bookworm”
root@rockpi4c:~#
root@rockpi4c:~# incus --version
0.1
root@rockpi4c:~#
root@rockpi4c:~# systemctl status incus.service
○ incus.service - Incus - Daemon
Loaded: loaded (/lib/systemd/system/incus.service; indirect; preset: enabled)
Active: inactive (dead)
TriggeredBy: ● incus.socket
root@rockpi4c:~#
root@rockpi4c:~# systemctl restart incus.service
root@rockpi4c:~# systemctl status -l incus.service
● incus.service - Incus - Daemon
Loaded: loaded (/lib/systemd/system/incus.service; indirect; preset: enabled)
Active: active (running) since Wed 2023-10-11 00:38:19 BST; 3s ago
TriggeredBy: ● incus.socket
Process: 12105 ExecStartPost=/opt/incus/lib/systemd/incusd waitready --timeout=600 (code=exited, status=0/SUCCESS)
Main PID: 12104 (incusd)
Tasks: 17
Memory: 26.5M
CPU: 1.353s
CGroup: /system.slice/incus.service
└─12104 incusd --group incus --logfile /var/log/incus/incusd.log

Oct 11 00:38:17 rockpi4c systemd[1]: Starting incus.service - Incus - Daemon…
Oct 11 00:38:17 rockpi4c incusd[12104]: time=“2023-10-11T00:38:17+01:00” level=warning msg=" - Couldn’t find the CGroup network priority controller, per-instance network prior>
Oct 11 00:38:17 rockpi4c incusd[12104]: time=“2023-10-11T00:38:17+01:00” level=warning msg=“Instance type not operational” driver=qemu err="vhost_vsock kernel module not loade>
Oct 11 00:38:18 rockpi4c incusd[12104]: time=“2023-10-11T00:38:18+01:00” level=warning msg="Firewall failed to detect any compatible driver, falling back to "xtables" (but s>
Oct 11 00:38:19 rockpi4c systemd[1]: Started incus.service - Incus - Daemon.
root@rockpi4c:~#
root@rockpi4c:~# grep incus /etc/group
incus:x:110:
incus-admin:x:111:
root@rockpi4c:~#
root@rockpi4c:~# usermod -aG incus-admin dietpi
root@rockpi4c:~#
root@rockpi4c:~# grep dietpi /etc/passwd
dietpi:x:1000:1000::/home/dietpi:/bin/bash
root@rockpi4c:~#
root@rockpi4c:~# grep dietpi /etc/group
dietpi:x:1000:
incus-admin:x:111:dietpi
root@rockpi4c:~#
-----

Thanks again!

I don’t know about LXDWare but yes, Incus and LXD can be co-installed on the same system.

Are they supposed to be able to co-exist naturally, or are some “hacks” required? Like installing the other under /usr and the other in /usr/local. Because I noticed that the fuidshift binary is offered by both incus and lxd and conflicts in your fs “naturally”. I can imagine you only need fuidshift from either and that it’d be compatible, but just throwing this observation out there.

EDIT: In fact I see fuidshift isn’t critical for either and could be shipped as a standalone tool. I think the official binaries for lxd or incus don’t include it?

That’s correct, fuidshift is just a demo/internal tool, it’s not used by either.
None of the main commands have conflicting names.

1 Like

Are fedora packages planned?

The Incus project itself doesn’t not produce packages, we leave that to community packagers and distributions to sort out as is customary with open source projects.

Currently the only packages we’re aware of are those that my company produces for Debian and Ubuntu users as well as the client-only packages for Windows and macOS.
There are a number of issues/package requests filed for a variety of distributions but it’s probably still a bit early to expect packages to be widely available.

A quick bit of Googling found an initial COPR repo for Incus on Fedora but packages aren’t yet available in it, so definitely looks like someone’s interested in getting that to work :slight_smile:

I will be interested in testing incus if it is anyday available on RockyLinux :slight_smile:

@stgraber Let me begin by saying that it appears that users are unable to create new topics under “incus” in this forum.

I month or so I indicated an addition that I wanted to see in LXD. It has not attracted any interest and so I thought that this might be an idea to be explored in the incus project. I have observed that there are OS images that do not seem to boot in incus VM’s.

Specifically, I am able to boot “tails-OS” (https://tails.net/) in a KVM in virt-manager. However, I am unable to boot the same OS image as an incus VM. The issue is that the TailsOS image requires that the boot device appear as being emulated as a USB device on a USB Disk bus. This is not the same as an actual USB device which is supported in an incus VM.

Here’s an example of a KVM VM with the configuration of a disk image that does work.

I have also determined that Android x86/64 images require the same “USB Bus” type in order to boot as well.

Does this sound like a feature that might be added to incus VM’s in the future?