IncusOS Clustering Question

IncusOS
1

I created 3 IncusOS VMs and I was trying to get them into a cluster. The instructions work for the first server, but I go off the rails for creating/adding the others.

From the tutorial (https://linuxcontainers.org/incus-os/docs/main/tutorials/incus-cluster/):

Adding additional servers

Additional servers need to use an installation image WITHOUT the default Incus settings enabled (Apply default configuration in the web-based image customizer).

That’s important as joining servers cannot have preexisting networks or storage pools defined, both of which get created as part of the default configuration.

Once the server is installed and added as a remote to the Incus command line tool using incus remote add, it can be added to the cluster with:

incus cluster join my-cluster: server2:
incus remote remove server2
incus cluster list my-cluster:

This will get server2 to join my-cluster, then remove the server-specific server2 remote and show an overview of the servers in the cluster.

If I create server using the ISO that does not have the “default” - the API doesn’t seem to be available on :8443 so I can’t add it as a remote from one of my other Incus hosts (where I do have access to the CLI, so not my first IncusOS VM).

I don’t get much further if I use an IncusOS VM that’s created with a “default” image. I can add it as a remote server, but then I find there is no “join” option for the incus cluster command? I’m using the LTS release on my “regular” hosts so maybe this is an option that’s not available there… But then I’m not too interested in creating a new Incus host where I do have a command line, just so I can use it as a remote to get the cluster going.

I’m obviously missing something.

2

Hmm, that’s pretty odd, we did have an issue where Incus wouldn’t listen on port 8443 unless the default settings were applied, but we fixed that quite a while back…

3

I reinstalled 3 physical servers just last week that all needed to join an existing cluster and I didn’t have any problem reaching 8443 on those. It’s certainly possible that we’ve regressed something since, but I’m not seeing any changes in that area.

4

Ah-ha! I downloaded a new customized image (so I got the 20260204 version). I re-imaged my VM and now the API is available on :8443. I guess whatever went wrong with the 20260203 update also caused this.

However, that still leaves the question how to join a remote incus host to a remote cluster (via a third-party incus host) if that host doesn’t provide the “incus cluster join” command?

5

You incus remote add both your existing cluster and your new server, then you use incus cluster join with both remotes.

incus cluster join existing-cluster: new-server:

6

Sorry - we’re not on the same wavelength. With my Incus version I get:

incus cluster join
Error: unknown command “join” for “incus cluster”

My install is from the zabbly Stable channel, version 6.15. Do I need a later version to make this work?

7

From my experience you should always use the latest incus version, currently 6.21.

IncusOS is still under heavy development and new features are added or changing with nearly every update. Only latest Incus clients will be compatible and support all IncusOS features.

Your version 6.15 was released way before IncusOS got stabilised.

In case you don’t want to update you can download the latest client binary from the github release page.

8

Thanks for the confirmation! I’ll step one of my non-IncusOS hosts up to the zabbly unstable channel and go from there.

9

you are on the correct channel “zabbly stable”, you perform an apt update, followed by apt upgrade incus. That should get you on latest 6.21. No need to change the apt source files.

10

Thanks, I’m having a senior moment. My 24.04 incus host upgraded fine to 6.21 … but my 22.04 doesn’t (should’ve realized I was on a older version of Ubuntu). Maybe 22.04 packages aren’t kept up-to-date anymore. Anyways, now I have an incus upgraded I can move to the next step.

1 Like
11

We do have 6.21 for Ubuntu 22.04 too, maybe it’s as simple as that system not having gotten the updated repository key? That would show up as an error during apt update.

12

Hey Stéphane - that is exactly what happened… I didn’t update the key for the repo :grimacing:

13

@stgraber @osch
I recreated my cluster VMs (IncusOS01, IncusOS02) from clean, using the 20260204 image. I didn’t create the third yet since I’m already running into issues.

IncusOS01 is built with the image with “Defaults” and IncusOS02 is built without defaults.

I enabled clustering on IncusOS01, and tried to join IncusOS02 (after adding it as a remote) with this result:

incus cluster join incus-cluster: IncusOS02:
What IP address or DNS name should be used to reach this server? [default=192.168.120.6]:
What member name should be used to identify this server in the cluster? [default=1ab68099-260f-4719-8c8b-a23ea0bcdc60]: IncusOS02
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose “source” property for storage pool “local”:
Choose “zfs.pool_name” property for storage pool “local”:
Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Failed to create storage pool “local”: Failed to run: zpool create -m none -O compression=on local /var/lib/incus/disks/local.img: exit status 1 (cannot create ‘local’: pool already exists)

Am I supposed to create all IncusOS instances without default configuration to be able to get the cluster to work?

14

Pass local/incus as the value for both source and zfs.pool_name.

15

Thanks, that did the trick! Maybe valuable to add to the tutorial for setting up the cluster with IncusOS?