I’m doing incus replications on my lab, 2 clusters, 3 nodes each. incus version 7.0.1. I use incus copy --refresh --stateless …
replication 1 : from sdcxlab02 > sdcxlab03 , same cluster, local, 10G network. copy to different incus project
replication 2 : from sdcxlab02 cluster1 to srdrlab02 cluster2. cluster 2 is remote DR cluster. 500 Mpbs network.
both replications are controlled by a lock file, both run in one cycle. no such thing as local and DR are trying to do incus copy the same instance simultaneously.
I run some containers and one vm1, nothing fancy on them, just generate 10MB file every 10 minutes. disks capacity are fine. the replication is on 10-minute cadence using cron.
now I get error like this below in my logfile, that happens only to vm1, all containers are ok. I find this on sdcxlab02, as source , where vm1 resides, error occured, something about /dev/zd272. all subsequent cycles are error.
root@sdcxlab02:~# cat /var/log/repl/repl-20260929.log | grep WARN
2026-09-29 02:50:44 [WARN] (247241) sdcxlab02 standard: default/vm1 (virtual-machine): DR refresh FAILED (exit=1) — Error: Failed instance creation: Error transferring instance data: Failed migration on target: Failed receiving migration offer from source: Failed migration on source: open /dev/zd272: no such device or address
2026-09-29 03:00:12 [WARN] (252324) sdcxlab02 standard: default/vm1 (virtual-machine): local refresh FAILED — Error: Error transferring instance data: Failed migration on target: Failed getting migration target filesystem connection: websocket: bad handshake
2026-09-29 03:00:12 [WARN] (252324) sdcxlab02 standard: default/vm1 (virtual-machine): DR refresh FAILED (exit=1) — Error: Failed migration on source: The instance is already being migrated
2026-09-29 03:10:10 [WARN] (254257) sdcxlab02 standard: default/vm1 (virtual-machine): local refresh FAILED — Error: Error transferring instance data: Failed migration on target: Failed getting migration target filesystem connection: websocket: bad handshake
2026-09-29 03:10:11 [WARN] (254257) sdcxlab02 standard: default/vm1 (virtual-machine): DR refresh FAILED (exit=1) — Error: Failed migration on source: The instance is already being migrated
2026-09-29 03:20:11 [WARN] (256318) sdcxlab02 standard: default/vm1 (virtual-machine): local refresh FAILED — Error: Error transferring instance data: Failed migration on target: Failed getting migration target filesystem connection: websocket: bad handshake
2026-09-29 03:20:12 [WARN] (256318) sdcxlab02 standard: default/vm1 (virtual-machine): DR refresh FAILED (exit=1) — Error: Failed migration on source: The instance is already being migrated
it started at 2026-09-29 02:50, it was ok at 02:40
the next error doesn’t say anything about any zd devices, just error like this :
root@sdcxlab02:~# /opt/repl/run-repl.sh standard
2026-09-30 13:42:56 [DEBUG] (412645) sdcxlab02 standard: Lock acquired: standard (PID 412645)
2026-09-30 13:42:56 [INFO] (412645) sdcxlab02 standard: === Cycle start ===
2026-09-30 13:42:57 [DEBUG] (412645) sdcxlab02 standard: Snapshot name: repl-20260930-1342
…
…
2026-09-30 13:42:57 [DEBUG] (412716) sdcxlab02 standard: default/vm1 (virtual-machine): CMD: incus copy vm1 vm1 --refresh --stateless --project default --target-project standby-default --target sdcxlab03
2026-09-30 13:42:57 [DEBUG] (412716) sdcxlab02 standard: default/vm1 (virtual-machine): CMD FAILED (exit=1): Error: Error transferring instance data: Failed migration on target: Failed getting migration target filesystem connection: websocket: bad handshake
2026-09-30 13:42:57 [WARN] (412716) sdcxlab02 standard: default/vm1 (virtual-machine): local refresh FAILED — Error: Error transferring instance data: Failed migration on target: Failed getting migration target filesystem connection: websocket: bad handshake
2026-09-30 13:42:57 [DEBUG] (412716) sdcxlab02 standard: default/vm1 (virtual-machine): DR refresh → srdrlab02:srdrlab02 (project default, timeout 900s)
2026-09-30 13:42:58 [WARN] (412716) sdcxlab02 standard: default/vm1 (virtual-machine): DR refresh FAILED (exit=1) — Error: Failed migration on source: The instance is already being migrated
…
2026-09-30 13:42:59 [INFO] (412716) sdcxlab02 standard: default/vm1 (virtual-machine): done (local=FAIL dr=FAIL)
2026-09-30 13:42:59 [INFO] (412645) sdcxlab02 standard: === Cycle complete (1 instances processed) ===
2026-09-30 13:42:59 [DEBUG] (412645) sdcxlab02 standard: Lock released: standard
on srdrlab02, which is the node2 on dr, I find something like this :
root@srdrlab02:~# incus operation list --format compact
ID TYPE DESCRIPTION STATE CANCELABLE CREATED LOCATION
2d6d9f46-e1e3-4bca-9dd7-7fbdbd96d177 TOKEN Executing operation RUNNING YES 2026/09/28 09:30 WIB srdrlab02
it’s clean on sdcxlab02 and sdcxlab03.
I try and delete the operation on srdrlab02, run the replication script again, and get the same error.
I can clear the error only after I do systemctl restart incus.service (and incus.socket) on sdcxlab02
several cycles go by and done ok, then it happens again. I do the incus restart again.
what is possibly the cause?