LXD/Incus compatibility

Incus
1

What are the goals for compatibility between LXD and incus? This will affect migrating from one to another.

I read here that incus is to have compatibility with LXD 5.16. But LXD is already at version 5.17.

Below I ask some compatibility questions and record what I’ve found by trying incus.

Images

I found that I can export images from LXD and import them to Incus, using commands like these:

  • lxc image export
  • incus image import .tar.gz

I assume it works in the other direction too, but I haven’t tried it. I hope images stay compatible for a long time.

Can we strive for a standard format for images that both LXD and Incus can be compatible with?

Image Servers

Can we add an LXD image server to an incus client configuration, or vice versa?
The default lxd and incus installations both include the simplestreams remote “images”,
and this server works with both LXD and Incus.

However, when I add add the “ubuntu” remote to incus, incus does not see any images in it.

API

Would I be able to build an executable with Go that works with both LXD and Incus?

I found that my existing programs that are compiled with the canonical LXD codebase mostly work with an incus server, and vice versa, provided I change the path for the UNIX socket to the correct one. I changed my code to try either the snap LXD socket or the incus socket, whichever exists.

But I encountered one operation that does not work properly, in either direction: When using InstanceServer.CreateInstanceFile(), InstanceFileArgs.Mode is ignored. Because of this one small problem, I cannot configure my incus containers with the same LXD-developed tools. I have to make changes and recompile incus-specific versions. From all the compatibility issues that I encountered, this is causing me the most trouble. Shall I report this as a bug?

command line

Can I run “lxc” commands with incus? It seems that the two clients have not diverged much yet, so I can add a link to incus called “lxc”:

sudo ln -s /usr/bin/incus /usr/local/bin/lxc

and my existing scripts will work with both LXD and incus.

miscellaneous

  • LXD uses group “lxd”, incus uses groups “incus”, “incus-admin”
  • LXD uses the default network lxdbr0, but incus uses incusbr0. This can be made compatible by naming the network bridge “lxdbr0” when running incus admin init.
  • LXD uses .lxd domain suffix for containers, incus uses .incus by default. They can be made the same using the command: incus network set lxdbr0 dns.domain lxd
  • LXD uses /var/snap/lxd/common/lxd/unix.socket. Incus uses /var/lib/incus/unix.socket.
  • LXD client uses ~/snap/lxd/common/config/ Incus client uses ~/.config/incus/
2

Yeah, we currently only support switching from LXD 5.16 or earlier, LXD LTS (5.0.x) being the most likely for production environments. LXD 5.17 can safely downgrade to 5.16, so that’s one way to switch to Incus.

Note that this will be changing as we get to pick up later changes from LXD into Incus.
We’re looking at supporting at least up to LXD 5.18 at this point.

Image format should be fine as they are pretty simple and the Incus team operates the main image server used by LXD too, so through that, we effectively control the image format and so can somewhat confidently say that LXD will keep compatible with Incus images.

This one is because of HTTP headers which differ between the two projects.
The client could probably be made to set both the LXD and Incus headers to accommodate that, though we obviously can’t guarantee that there won’t be more complex API breakages down the line.

For the lxd-to-incus migration tool, we import both the incus and lxd clients in the same binary, specifically so we don’t rely on the incus client to talk to LXD or vice-versa.

No, you should definitely not be doing that.
The two command line tools have diverged significantly already and will likely diverge more in the future. incus snapshot, incus admin, incus config trust add and incus cluster add behave completely differently from LXD’s version.

3

Forgot to answer that general question.

The goal is to allow for easy migration of existing LXD users over to Incus by importing their data through a migration tool. This will have to be restricted to select versions of LXD though with users on unsupported versions having to do it the hard way through exporting instances or the like.

The command line tools are not identical and so should not be aliased for the other.
The API is not strictly identical, features have been removed and specific endpoints have had to be updated. It is however compatible enough that moving instances between the two should be fine for now.

Ideally we’ll try to support moving from LXD 5.0.x and LXD 5.x to Incus, though the last few 5.x releases may get tricky (pre 6.0).

After 6.0, I don’t expect we’ll be able to support easy in-place import from LXD to Incus, instead requiring each individual instance be exported and then manually imported. That’s because as soon as we start getting diverging database schemas between LXD and Incus, the effort needed for a safe conversion will grow exponentially.

1 Like
4

Is this a decision still yet to be made or are you pretty confident in 5.18 support for migration? I’m curious when to stop upgrading my own machines and also thinking through how to provide a good transition path for NixOS packaging purposes. I had missed the 5.16 mention before, so NixOS unstable already has 5.18 and it’ll be in the next stable (23.11).

5

I haven’t reviewed 5.18 yet, for 5.17 we know it’s fine as there’s no DB change from 5.16.
Hopefully it’s the same story with 5.18 or it’s DB changes that we do want in Incus.

1 Like
6

Hope it’s ok to revive this thread since my question is related: is it possible to install incus alongside lxd (and have containers running under both) to ease the migration?

7

Yep, that works perfectly fine in my experience.

1 Like
8

Just to add for reference: I’ve used the script to migrate from lxd 5.20 to incus 0.6 without issues (except it asked me to unset core.trust_password first).

On other systems I’ve now done snap refresh --hold=forever lxd to ensure that lxd stays on 5.20, to keep my options open for a future migration.