Incusos 6.20 - can't join cluster

1

I’m trying to cluster together a couple of nodes in my homelab. After migrating all workloads to the cluster leader (incus2.x.family), I’m trying to join incus3 to incus2. I’m getting errors that the storage pool keys are cluster node specific.

 ~ % incus cluster list incus2.x.family
±-------±----------------------------±----------------±-------------±---------------±------------±-------±------------------+
|  NAME  |             URL             |      ROLES      | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATUS |      MESSAGE      |
±-------±----------------------------±----------------±-------------±---------------±------------±-------±------------------+| 
incus2 | https://192.168.50.102:8443 | database-leader | x86_64       | default        |             | ONLINE | Fully operational ||        |                             | database        |              |                |             |        |                   |
±-------±----------------------------±----------------±-------------±---------------±------------±-------±------------------+

~ % incus cluster join incus2.x.family: incus3.x.family: What IP address or DNS name should be used to reach this server? [default=192.168.50.103]: What member name should be used to identify this server in the cluster? [default=4e4e7f80-19f3-11f0-b0d1-5cf6c0f4bf07]: incus3 All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes Choose “source” property for storage pool “local”: local/incus Choose “zfs.pool_name” property for storage pool “local”: local/incus Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Failed to update storage pool “local”: Config key “source” is cluster member specific

 ~ % incus storage ls
±------±-------±-------------------------------------±--------±--------+
| NAME  | DRIVER |             DESCRIPTION              | USED BY |  STATE  |
±------±-------±-------------------------------------±--------±--------+
| local | zfs    | Local storage pool (on system drive) | 3       | CREATED |
±------±-------±-------------------------------------±--------±--------+

Also tested removing as many resources as possible, but the pool doesn’t seem to be removable:

~ % incus profile device remove default root
incus profile device remove default eth0
incus network delete incusbr0
incus storage delete local
Device root removed from default
Device eth0 removed from default
Network incusbr0 deleted
Error: The storage pool is currently in use

Afterwards I’m still not able to join the cluster sadly. Any help would be greatly appreaciated!

2

Indeed you’ll need to delete everything, especially the pool and network.
You can do incus storage show local to see what’s using the pool.

Most likely it’s the local configuration for the storage and backup volumes:

incus config unset storage.backups_volume
incus config unset storage.images_volume
incus storage volume delete local backups
incus storage volume delete local images
incus storage delete local
3

Thanks so much for your quick reply stgraber. That was indeed the part that I was missing. I think it would do nicely in the incusos manual as well :slight_smile:

4

Yeah, we need to add a tutorial on clustering to the docs.

For setting up a new environment with IncusOS, your best bet is:

  • Grab a first ISO with the “Apply defaults” option enabled, use that for the first cluster server
  • Grab a second ISO with the “Apply defaults” option disabled and use that for any other server

That way you don’t need to start deleting things to get the other servers to join.