Incus copy struck after cancellation of source operation

Incus
1

May be unrelated, if you “incus copy” a container between two nodes and cancel in the GUI mid-way, it leaves a ghost on the target. If you then try to repeat the copy, it errors with;

Error: Failed migration on source: Error from migration control target: Failed \
     creating instance on target: Volume exists in database but not on storage

If you delete the ghost and retry it works.
Sample copy command;

incus copy ${host}:${inst} $inst --target-project=${project} \
                             --refresh --stateless --target=${self}

My expectation is that if a copy is cancelled, the partial would be deleted from the target. Either way, a repeat copy should subsequently work rather than erroring …

2

Copies aren’t a cancellable operation type, so not sure what the UI cancelled exactly :slight_smile:

3

Copies aren’t a cancellable operation type, so not sure what the UI cancelled exactly 

Well, I seem to have a flair for the not-possible. I can send you a screen grab if you like, but this is a copy / paste from my operations log;

* Success

Operation Migrating instance cancelled

Close notification

29/06/2025, 21:50:39

But the fact remains, it is possible to do a copy between machines, for the copy to fail for whatever reason, and for that failure to block a repeat of the copy. In a backup / --refresh scenario this breaks backups.

4

So, my day so far has been trying to complete a backup. It would appear the situation is as follows;

As a result of “other” issues, my backup node ran out of space, so one relatively large backup (30G) failed. I’ve now been through many iterations of retries and trying to find out why repeats won’t work and I appear to have arrived at the following;

  • There is a stuck operation on the source node, action = “Migrating instance”
  • Source and destination incus daemons have been restarted, so nothing is happening
  • There appears to be no way to cancel the copy
  • The copy is blocking new copy attempts
  • Net result is that I can’t get a backup copy

Last copy command, i.e. the one that looks stuck;

incus copy worf:email-aaa email-aaa --target-project=backup --target=rad --mode=push

Previous attempts include mode=pull, --stateless, --refresh

I guess I need to find and clear the sql operations table … (?)

5

So … kill the stuck “rsync” (no idea why it’s using rsync when it should be using zfs send), delete * from operations, kill -9 on incusd, seems to have cleared it … copy now in progress. Again it’s using rsync.