Issues joining a server to an existing cluster (UB24.04 to UB22.04 Cluster)

1

Hello, i’m facing a little issue, but not sure if it’s a bug, before filling an issue on Github, i thought it would be beter to ask about that around here :slight_smile:

I’m running a 3 node cluster, to which i’d like to add another node :slight_smile:

Node1,2 and 3 runs on Ubuntu 22.04, Node4 runs on Ubuntu 24.04.

I’m pretty used to the joining process, everything goes fine, until i get this error :

Error: Failed to join cluster: Failed request to add member: The joining server version doesn't match (expected 6.0.2 with API count 418)

Do every node in a cluster must run the same OS version ?

During joining operation, storage pools are created without errors :slight_smile:

Node1 : incus version
Client version: 6.0.2
Server version: 6.0.2
lsb_release -a : 
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.5 LTS
Release:	22.04
Codename:	jammy



Node4 : incus version
Client version: 6.0.2
Server version: 6.0.2
lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 24.04.1 LTS
Release:	24.04
Codename:	noble

I’m using Zabbly pakages repository on both systems (added to 22.04 and 24.04) :

Node1 (22.04) :  $ apt info incus -a
Package: incus
Version: 1:6.0.2-202409162053-ubuntu22.04
Priority: optional
Section: admin
Maintainer: Zabbly ZFS builds <info@zabbly.com>
Installed-Size: 75.0 MB
Depends: incus-base (= 1:6.0.2-202409162053-ubuntu22.04), genisoimage, libnuma1, libpixman-1-0, librados2, librbd1, libseccomp2, libspice-server1, libusb-1.0-0, libusbredirparser1 | libusbredirparser1t64
Homepage: https://linuxcontainers.org
Download-Size: 26.3 MB
APT-Manual-Installed: yes
APT-Sources: https://pkgs.zabbly.com/incus/lts-6.0 jammy/main amd64 Packages
Description: Incus - Container and virtualization daemon
 Incus provides the ability to run containers and virtual machines locally
 along with very flexible networking and storage akin to what's offered
 in a public cloud environment.
 .
 This package contains everything needed to run containers and VMs.
 .
 This build of Incus is provided for free by Zabbly.
 Zabbly can be contacted for commercial support and development on Incus.

Package: incus
Version: 1:6.0.1-202409022014-ubuntu22.04
Priority: optional
Section: admin
Maintainer: Zabbly ZFS builds <info@zabbly.com>
Installed-Size: 72.8 MB
Depends: incus-base (= 1:6.0.1-202409022014-ubuntu22.04), genisoimage, libnuma1, libpixman-1-0, librados2, librbd1, libseccomp2, libspice-server1, libusb-1.0-0, libusbredirparser1 | libusbredirparser1t64
Homepage: https://linuxcontainers.org
Download-Size: 25.6 MB
APT-Sources: https://pkgs.zabbly.com/incus/lts-6.0 jammy/main amd64 Packages
Description: Incus - Container and virtualization daemon
 Incus provides the ability to run containers and virtual machines locally
 along with very flexible networking and storage akin to what's offered
 in a public cloud environment.
 .
 This package contains everything needed to run containers and VMs.
 .
 This build of Incus is provided for free by Zabbly.
 Zabbly can be contacted for commercial support and development on Incus.


Node04 (24.04) : apt info incus -a
Package: incus
Version: 1:6.0.2-202409162106-ubuntu24.04
Priority: optional
Section: admin
Maintainer: Zabbly ZFS builds <info@zabbly.com>
Installed-Size: 75.1 MB
Depends: incus-base (= 1:6.0.2-202409162106-ubuntu24.04), genisoimage, libnuma1, libpixman-1-0, librados2, librbd1, libseccomp2, libspice-server1, libusb-1.0-0, libusbredirparser1 | libusbredirparser1t64
Homepage: https://linuxcontainers.org
Download-Size: 26.2 MB
APT-Manual-Installed: no
APT-Sources: https://pkgs.zabbly.com/incus/lts-6.0 noble/main amd64 Packages
Description: Incus - Container and virtualization daemon
 Incus provides the ability to run containers and virtual machines locally
 along with very flexible networking and storage akin to what's offered
 in a public cloud environment.
 .
 This package contains everything needed to run containers and VMs.
 .
 This build of Incus is provided for free by Zabbly.
 Zabbly can be contacted for commercial support and development on Incus.

Package: incus
Version: 1:6.0.1-202409022024-ubuntu24.04
Priority: optional
Section: admin
Maintainer: Zabbly ZFS builds <info@zabbly.com>
Installed-Size: 72.9 MB
Depends: incus-base (= 1:6.0.1-202409022024-ubuntu24.04), genisoimage, libnuma1, libpixman-1-0, librados2, librbd1, libseccomp2, libspice-server1, libusb-1.0-0, libusbredirparser1 | libusbredirparser1t64
Homepage: https://linuxcontainers.org
Download-Size: 25.4 MB
APT-Sources: https://pkgs.zabbly.com/incus/lts-6.0 noble/main amd64 Packages
Description: Incus - Container and virtualization daemon
 Incus provides the ability to run containers and virtual machines locally
 along with very flexible networking and storage akin to what's offered
 in a public cloud environment.
 .
 This package contains everything needed to run containers and VMs.
 .
 This build of Incus is provided for free by Zabbly.
 Zabbly can be contacted for commercial support and development on Incus.

Package: incus
Version: 1:6.0.1-202408280413-ubuntu24.04
Priority: optional
Section: admin
Maintainer: Zabbly ZFS builds <info@zabbly.com>
Installed-Size: 72.9 MB
Depends: incus-base (= 1:6.0.1-202408280413-ubuntu24.04), genisoimage, libnuma1, libpixman-1-0, librados2, librbd1, libseccomp2, libspice-server1, libusb-1.0-0, libusbredirparser1 | libusbredirparser1t64
Homepage: https://linuxcontainers.org
Download-Size: 25.4 MB
APT-Sources: https://pkgs.zabbly.com/incus/lts-6.0 noble/main amd64 Packages
Description: Incus - Container and virtualization daemon
 Incus provides the ability to run containers and virtual machines locally
 along with very flexible networking and storage akin to what's offered
 in a public cloud environment.
 .
 This package contains everything needed to run containers and VMs.
 .
 This build of Incus is provided for free by Zabbly.
 Zabbly can be contacted for commercial support and development on Incus.

Package: incus
Version: 6.0.0-1ubuntu0.2
Built-Using: go-md2man-v2 (= 2.0.3+ds1-1ubuntu0.1), [...] gopacket (= 1.1.19-6)
Priority: optional
Section: universe/admin
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian Go Packaging Team <team+pkg-go@tracker.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 59.7 MB
Depends: systemd | systemd-standalone-sysusers | systemd-sysusers, libacl1 (>= 2.2.23), libc6 (>= 2.38), libcap2 (>= 1:2.10), libcowsql0, libgcc-s1 (>= 3.3.1), liblxc1t64 (>= 1:4.0.2), libsqlite3-0 (>= 3.36.0), libudev1 (>= 199), attr, ca-certificates, liblxc-common, lxcfs, incus-client (= 6.0.0-1ubuntu0.2), iproute2, nftables | iptables, rsync, squashfs-tools, uidmap, xz-utils
Recommends: apparmor, dnsmasq-base, incus-agent
Suggests: btrfs-progs, ceph-common, lvm2, zfsutils-linux, incus-tools, gdisk
Homepage: https://linuxcontainers.org/incus/
Download-Size: 18.2 MB
APT-Sources: http://gf.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages
Description: Powerful system container and virtual machine manager - daemon
 Incus is a next generation system container and virtual machine manager.
 It offers a unified user experience around full Linux systems running
 inside containers or virtual machines.
 .
 It's image based with pre-made images available for a wide number of
 Linux distributions and is built around a very powerful, yet pretty
 simple, REST API.
 .
 This package contains the Incus daemon.

Package: incus
Version: 6.0.0-1
Built-Using: go-md2man-v2 (= 2.0.3+ds1-1), [...] gopacket (= 1.1.19-6)
Priority: optional
Section: universe/admin
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian Go Packaging Team <team+pkg-go@tracker.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 59.7 MB
Depends: systemd | systemd-standalone-sysusers | systemd-sysusers, libacl1 (>= 2.2.23), libc6 (>= 2.38), libcap2 (>= 1:2.10), libcowsql0, libgcc-s1 (>= 3.3.1), liblxc1t64 (>= 1:4.0.2), libsqlite3-0 (>= 3.36.0), libudev1 (>= 199), attr, ca-certificates, liblxc-common, lxcfs, incus-client (= 6.0.0-1), iproute2, nftables | iptables, rsync, squashfs-tools, uidmap, xz-utils
Recommends: apparmor, dnsmasq-base, incus-agent
Suggests: btrfs-progs, ceph-common, lvm2, zfsutils-linux, incus-tools, gdisk
Homepage: https://linuxcontainers.org/incus/
Download-Size: 18.2 MB
APT-Sources: http://gf.archive.ubuntu.com/ubuntu noble/universe amd64 Packages
Description: Powerful system container and virtual machine manager - daemon
 Incus is a next generation system container and virtual machine manager.
 It offers a unified user experience around full Linux systems running
 inside containers or virtual machines.
 .
 It's image based with pre-made images available for a wide number of
 Linux distributions and is built around a very powerful, yet pretty
 simple, REST API.
 .
 This package contains the Incus daemon.

I bieleve both packets are pretty the same, i first thought i’d be running in an issue with ‘Ubuntu’ packages, but as they did not update to 6.0.2, i bieleve it’s not the case.

I don’t know if some update has been commited in the time frame between :
202409162053 (22.04) and 202409162106 (24.04) ?

Thanks in advance for any advice… :slight_smile:

It would be nice to be able to run incus across diffrent versions of the same OS, for updating purposes :slight_smile:

2

Can you show incus admin sql global "SELECT * FROM nodes" both from an existing clustered server and from the joining one?

3

Hello, thanks for your help :slight_smile:

On existing node member :

±—±--------------------±------------±------------------±-------±---------------±-------------------------------±------±-----±------------------+
| id | name | description | address | schema | api_extensions | heartbeat | state | arch | failure_domain_id |
±—±--------------------±------------±------------------±-------±---------------±-------------------------------±------±-----±------------------+
| 1 | cloud-incus-02-bkra | | 10.75.13.252:8444 | 73 | 418 | 2024-11-27T04:06:42.319317519Z | 0 | 2 | |
| 2 | cloud-incus-01-leo | | 10.75.13.253:8444 | 73 | 418 | 2024-11-27T04:06:43.324892659Z | 0 | 2 | |
| 3 | cloud-incus-03-oth | | 10.75.13.251:8444 | 73 | 404 | 2024-08-16T01:05:02.50886685Z | 2 | 2 | |
±—±--------------------±------------±------------------±-------±---------------±-------------------------------±------±-----±------------------+

On the new node member :

±—±-----±------------±--------±-------±---------------±---------------------±------±-----±------------------+
| id | name | description | address | schema | api_extensions | heartbeat | state | arch | failure_domain_id |
±—±-----±------------±--------±-------±---------------±---------------------±------±-----±------------------+
| 1 | none | | 0.0.0.0 | 73 | 418 | 2024-11-27T02:44:21Z | 0 | 2 | |
±—±-----±------------±--------±-------±---------------±---------------------±------±-----±------------------+

I Bieleve it’s because an ‘old’ node is outdated !
I’ll update it, and see if it solves the problem :slight_smile:

Thanks a lot for your help :slight_smile:

4

Thanks again Stephane,

Straight to the point !

A little bit of context : Node3 is actually evacuated, and served solely as a backup node… as i added a backup plan offsite, it was taken offline from late august to cut powerdraw :slight_smile: i did not took care to update it when Incus 6.0.2 went out :slight_smile:

I just added this node flawlessly, it took a little bit of time, i bieleve some images / custom iso got copied to it, and as it’s a very stretched cluster (FR / Germany / French Guyana), took a little bit of time to transfer data :slight_smile:

It now shows up in incus :

+---------------------+---------------------------+------------------+--------------+----------------+-------------+-----------+--------------------------------+
|        NAME         |            URL            |      ROLES       | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION |  STATUS   |            MESSAGE             |
+---------------------+---------------------------+------------------+--------------+----------------+-------------+-----------+--------------------------------+
| cloud-incus-01-leo  | <redacted>  | database         | x86_64       | default        |             | ONLINE    | Fully operational              |
+---------------------+---------------------------+------------------+--------------+----------------+-------------+-----------+--------------------------------+
| cloud-incus-02-bkra | <redacted>  | database-leader  | x86_64       | default        |             | ONLINE    | Fully operational              |
|                     |                           | database         |              |                |             |           |                                |
+---------------------+---------------------------+------------------+--------------+----------------+-------------+-----------+--------------------------------+
| cloud-incus-03-oth  | <redacted>  | database         | x86_64       | default        |             | EVACUATED | Unavailable due to maintenance |
+---------------------+---------------------------+------------------+--------------+----------------+-------------+-----------+--------------------------------+
| cloud-incus-04-slm  | <redacted> | database-standby | x86_64       | default        |             | ONLINE    | Fully operational              |
+---------------------+---------------------------+------------------+--------------+----------------+-------------+-----------+--------------------------------+

+----+---------------------+-------------+-------------------+--------+----------------+--------------------------------+-------+------+-------------------+
| id |        name         | description |      address      | schema | api_extensions |           heartbeat            | state | arch | failure_domain_id |
+----+---------------------+-------------+-------------------+--------+----------------+--------------------------------+-------+------+-------------------+
| 1  | cloud-incus-02-bkra |             | 10.75.13.252:8444 | 73     | 418            | 2024-11-27T05:07:32.563260198Z | 0     | 2    | <nil>             |
| 2  | cloud-incus-01-leo  |             | 10.75.13.253:8444 | 73     | 418            | 2024-11-27T05:07:37.410535045Z | 0     | 2    | <nil>             |
| 3  | cloud-incus-03-oth  |             | 10.75.13.251:8444 | 73     | 418            | 2024-11-27T05:07:34.679951625Z | 2     | 2    | <nil>             |
| 4  | cloud-incus-04-slm  |             | 10.75.13.250:8443 | 73     | 418            | 2024-11-27T05:07:39.928711799Z | 0     | 2    | <nil>             |
+----+---------------------+-------------+-------------------+--------+----------------+--------------------------------+-------+------+-------------------+

Thanks a lot :slight_smile:

Best regards,