I’ve had this happen on a regular incus server where a container move fails, say due to criu (and my forgetting to shut down the source container), then subsequent attempts to move the instance fail with: Error: Failed migration on source: Error from migration control target: Failed creating instance on target: Volume already exists on storage but not in database
With regular incus I know to look for and manually destroy the errant ZFS dataset on the target that did not get cleaned up after failure. However, with IncusOS I’m at a loss as to how to see let alone clean up such detritis.
If you have a reproducer for this, we’d love to fix the issue in Incus on that one.
To fix the issue on the IncusOS box, there are two main ways you can do it, both rather ugly:
Create a privileged container, pass /dev/zfs as a unix-char device and install the ZFS tools in the container, that should then let you see the storage pool and run zfs destroy on it
Re-create the missing database record through incus admin sql global which would then let you perform the deletion, cleaning up the ZFS volume and any on-disk structure
Thanks for the recovery steps. I’ll try the privileged container method since using the ZFS tools is more familiar than doing DB surgery (although I have seen you reply in a similar vein before and can find the details if I need to try that method).
I was able to create a privileged Debian container (targeting the same IncusOS cluster member that the original move fail to move to), add /dev/zfs as a character device with default values (used WebUI), install zfsutils-linux in the privileged container, see that /dev/zfs was dynamically created, run zfs list to see the dataset that was obstructing the move, and finally run zfs destroy local/incus/containers/lldap1.
After that I was able to move the container no problem incus move p1:lldap1 incusos-v10:lldap1 --target incusos-v10 --stateless
For a reproducer, I’m not sure if you prefer/need the source and target Incus servers from scratch or can I directly test with a new container in my existing clusters. My source cluster is 3 Incus on Debian-13 physical machines. My target cluster is Incus on IncusOS virtual machines (hosted on the physical cluster).
I’m convincing myself I should spin up two test clusters similar to my pair above, except both virtual. I need to play/learn to use tools like opentofu anyway, so that will be a motivation.
I will say that the existence of snapshots might be a factor based on preliminary tests. I think it might also help to have a non-trivial workload in the container to make criu struggle more.