I was planning to move my cluster interfaces to dedicated non-routable vlan and found that the cluster.http_address' value has been changed to the 'core.http_address value when I modified cluster node IPs using incus admin cluster edit. So this change blocked access to the API from remote clients. I tied to change config option cluster.http_address but got the error: Error: Changing cluster.https_address is currently not supported..
Comment in the source code says: We currently don't allow changing the cluster.https_address once it's set.. However, I did not intend to change cluster.http_address when I changed cluster nodes IPs, but rhe incus did this for me. Is there any way to safely move cluster nodes to another subnet without changing the cluster.http_address ?
Ok, this is just from observation so someone please correct me where I’m wrong …
As I understand it, the cluster.https_address is what the cluster nodes use to talk to each other, and the core.https_address is the address the UI / API will listen on. When you configure the cluster, you tell it which interface / address to listen on, which should set the cluster.https_address.
If you were to change this, you would need to change all three nodes as the same time, which in itself might prove problematic. After working with OVN recently, if Incus’s Raft is anything like OVN, it will not be happy with IP changes and break horribly. So even if you could, it might not be a good idea to try …
I recently was faced with the same issue. My solution was to copy all instances to an incus archive node, set up a new cluster from scratch on the desired IP addresses, then copy the instances from the archive node back to the new cluster. In reality I had two clusters running in parallel and subtracted a node from one then added to the other. Process worked Ok, albeit a little time consuming.
If it helps, I use this to backup all my nodes every hour or so, same principle to archive everything off to another node;
#!/usr/bin/env bash
self=borg # the node we're running on
project=cluster # the backup projec to take the copies
echo --- Storage --------- $(date) ------------
for i in `incus storage volume ls local -c nL -fcsv type=custom|grep -v "/"`
do
inst=`echo $i|cut -d"," -f1`
host=`echo $i|cut -d"," -f2`
echo "Copying custom (docker) Image :: ${inst} from ${host}"
incus storage volume copy local/${inst} ${self}:local/${inst} --target-project=${project} --refresh
done
echo --- Containers --------- $(date) ------------
for i in `incus list -c nL -f csv,noheader|tr ' ' '_'`
do
inst=`echo $i|cut -d"," -f1`
host=`echo $i|cut -d"," -f2`
echo "Copying :: ${inst} from ${host} to ${self}"
incus copy $inst ${self}:$inst --target-project=${project} --refresh --stateless
done
echo ------------ $(date) ------------
It’s not impossible to change it, but it’s a bit involved…
You basically need to:
Completely stop Incus on all servers (both incus.service and incus.socket)
Write a /var/lib/incus/database/patch.global.sql to change the address in the nodes table
Modify /var/lib/incus/database/local.db on all servers for the new address
Run incus admin cluster edit on all servers to change the address
Start Incus back on all servers
I’ve done it before in some situations for some customers, but it’s definitely not a particularly smooth or downtime-free process which is why we don’t have a way to do it online through the API.
Wouldn’t this work as a (mostly, modulo the time needed to migrate an instance) downtime-free variant:
evacuate one cluster node in the old cluster
remove that node from the cluster and wipe the incus configuration on that node
create a new cluster on that node with the changed address
start the new cluster (even if it just has one node yet)
move the evacuated vms/containers in the old cluster over to the new cluster (should not overload the node because it is exactly the same load as before)
repeat until all nodes are migrated
evacuate another node in the old cluster
stop incus, wipe config, add node to the new cluster
move the evacuated instances of the old cluster to the new node
Fortunately, this happened on staged cluster and I did not lost any data, except my time
I agree with @oddjobz and @rkunze that the safest way to changing cluster nodes IP is the removing node form the cluster->changing IP->joining node to the cluster operation.
However, the real problem is the unexpected behavior. This part is not clearly documented and it does not say that the cluster.http_address is always the node address defined in incus admin init and changing the node address will automatically change API listener on this node.
P.S.
In fact I have to rebuild my staged cluster due to lost incus synchronization with the OVN. The Incus daemons did stuck on start and ovs-central flooded with warnings about non-exist bridges created by the incus in the past.
Yeah, I’ve seen a lot of this over the last month or so. Once your OVN raft is broken the only obvious way to recover is delete the data and start again. Then you’ve got the problem of Incus ovn objects being out of sync.
However
In many cases I’ve found that the ovn database “would” be fine if only the nodes would sync with each other, and doing things like changing the node address, or maybe reinstalling a couple of nodes without deleting them first hence leaving phantoms and losing quorum, it’s very easy to do in testing. I did eventually discover it’s relatively easy to recover this (!)
(make backups, adjust paths, ports etc for your distro, this is debian)
Close down all but one node
Delete /var/lib/ovn/* and /var/lib/openvswitch/* # on all but one node
Restart other nodes, assuming you supply host and port params on the command line rather than relying on the DB, they should just connect up as they would on a new db
In this instance you don’t lose sync with Incus which makes recovery Soooo much easier.
Everything looks good, however, nodes lost communication to each other.
Running the incus daemon in debug mode I see that the node tries to use use cluster.http_address instead of core.http_address as a node IP to handover member role and fails.
INFO [2025-06-12T18:04:44Z] Starting up mode=normal path=/var/lib/incus version=6.13
INFO [2025-06-12T18:04:44Z] Current process idmap:
INFO [2025-06-12T18:04:44Z] - u 0 0 4294967295
INFO [2025-06-12T18:04:44Z] - g 0 0 4294967295
INFO [2025-06-12T18:04:44Z] Selected idmap:
INFO [2025-06-12T18:04:44Z] - u 0 1000000 1000000000
INFO [2025-06-12T18:04:44Z] - g 0 1000000 1000000000
INFO [2025-06-12T18:04:44Z] Kernel features:
INFO [2025-06-12T18:04:44Z] - closing multiple file descriptors efficiently: yes
INFO [2025-06-12T18:04:44Z] - netnsid-based network retrieval: yes
INFO [2025-06-12T18:04:44Z] - pidfds: yes
INFO [2025-06-12T18:04:44Z] - pidfds for threads: no
INFO [2025-06-12T18:04:44Z] - core scheduling: yes
INFO [2025-06-12T18:04:44Z] - uevent injection: yes
INFO [2025-06-12T18:04:44Z] - seccomp listener: yes
INFO [2025-06-12T18:04:44Z] - seccomp listener continue syscalls: yes
INFO [2025-06-12T18:04:44Z] - seccomp listener add file descriptors: yes
INFO [2025-06-12T18:04:44Z] - attach to namespaces via pidfds: yes
INFO [2025-06-12T18:04:44Z] - safe native terminal allocation: yes
INFO [2025-06-12T18:04:44Z] - unprivileged binfmt_misc: yes
INFO [2025-06-12T18:04:44Z] - unprivileged file capabilities: yes
INFO [2025-06-12T18:04:44Z] - cgroup layout: cgroup2
INFO [2025-06-12T18:04:44Z] - idmapped mounts kernel support: yes
INFO [2025-06-12T18:04:44Z] Instance type operational driver=lxc type=container
WARNING[2025-06-12T18:04:44Z] Instance type not operational driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine
INFO [2025-06-12T18:04:44Z] Initializing local database
DEBUG [2025-06-12T18:04:44Z] Refreshing local trusted certificate cache
INFO [2025-06-12T18:04:44Z] Set client certificate to server certificate fingerprint=ebfde364a490746bf66cddc0d7b3709b105a7cc07edd61e4f03141281ddea67e
DEBUG [2025-06-12T18:04:44Z] Initializing database gateway
INFO [2025-06-12T18:04:44Z] Loading daemon configuration
INFO [2025-06-12T18:04:44Z] Binding socket socket="192.168.182.33:8443" type="cluster socket"
INFO [2025-06-12T18:04:44Z] Binding socket socket=/var/lib/incus/guestapi/sock type="devIncus socket"
INFO [2025-06-12T18:04:44Z] Binding socket socket="172.16.120.3:8443" type="REST API TCP socket"
INFO [2025-06-12T18:04:44Z] Binding socket socket="192.168.182.33:8443" type="cluster socket"
INFO [2025-06-12T18:04:44Z] Binding socket socket=/var/lib/incus/unix.socket type="REST API Unix socket"
INFO [2025-06-12T18:04:44Z] Initializing global database
INFO [2025-06-12T18:04:44Z] Connecting to global database
DEBUG [2025-06-12T18:04:44Z] Dqlite connected outbound local="172.16.120.3:33864" name=dqlite remote="172.16.120.1:8443"
DEBUG [2025-06-12T18:04:44Z] Dqlite: attempt 1: server 172.16.120.1:8443: connected
INFO [2025-06-12T18:04:44Z] Connected to global database
DEBUG [2025-06-12T18:04:44Z] Database error err="Failed to update cluster member version info: updated 0 rows instead of 1"
ERROR [2025-06-12T18:04:44Z] Failed to start the daemon err="Failed to initialize global database: failed to ensure schema: Failed to update cluster member version info: updated 0 rows instead of 1"
INFO [2025-06-12T18:04:44Z] Starting shutdown sequence signal=interrupt
INFO [2025-06-12T18:04:44Z] Handing over cluster member role address="192.168.182.33:8443"
DEBUG [2025-06-12T18:04:44Z] Connecting to a remote Incus over HTTPS url="https://172.16.120.1:8443"
DEBUG [2025-06-12T18:04:44Z] Sending request to Incus etag= method=POST url="https://172.16.120.1:8443/internal/cluster/handover"
DEBUG [2025-06-12T18:04:44Z]
{
"address": "192.168.182.33:8443"
}
WARNING[2025-06-12T18:04:44Z] Could not handover member's responsibilities err="No dqlite node has address 192.168.182.33:8443: %!w(<nil>)"
DEBUG [2025-06-12T18:04:44Z] Cancel ongoing or future gRPC connection attempts
DEBUG [2025-06-12T18:04:44Z] Cancel ongoing or future gRPC connection attempts
DEBUG [2025-06-12T18:04:44Z] Stop database gateway```
