Moving ZFS based container to another pool efficiently

Incus
,
1

Hi!

Recently I’ve added a new big storage device to my Incus host and I’d like to move my container instances to that new storage. Both storages are ZFS-based, so I’d like to make use of the ZFS sync feature. As this comment by @tomp states, in this case, ZFS sync should be used in the background if I utilize the “incus move -s …” command.

That is fine, but with a 300+ GB container, it will certainly take a while. I was wondering if there is a better way. I think of something like that:

  1. Taking a snapshot of the old storage while the container is running
  2. “zfs sync” that snapshot to the new storage
  3. When it’s finished, stop the container, transfer diff to new storage via zfs sync -i (incremental).
  4. Switch container to new storage, start it.
  5. (delete snapshot of old storage)

This would drastically reduce downtime, as the vast majority of data can be synced in the background without affecting the running container. Only during the finishing diff sync (ZFS incremental sync) there will be a container downtime of few seconds.

Regarding this way of moving the container data, I have two questions:

  1. Has this been considered to be a supported way of moving container data? e.g. by an “incus move –prepare -s mynewstorage” command and an “incus move –finish” command, that only transfers the diff very quickly? I’d love to see this feature! :heart_eyes:
  2. I guess this process can be done manually while working around Incus a bit. I could create a ZFS snapshot and sync manually by using ZFS commands. But what about the Incus part? Probably I need to manipulate the Incus database to point to the new storage, right? Does this create any pitfalls or do you have hints how to “zfs sync -i” without causing any trouble to Incus?

Thank you very much! Incus is one of my very favorite tools! :+1:

2

You should be able to do:

  • incus snapshot create NAME
  • incus copy NAME tmp-NAME --storage NEW-STORAGE
  • incus snapshot create NAME
  • incus copy NAME tmp-NAME --storage NEW-STORAGE --refresh
  • incus stop NAME
  • incus copy NAME tmp-NAME --storage NEW-STORAGE --refresh
  • incus delete NAME
  • incus move tmp-NAME NAME
  • incus start NAME
2 Likes
3

Sounds good, thank you! I’ll try these commands and let you know if they worked for me. :slight_smile:

4

You probably will want to do a incus config unset NAME volatile.apply_template before that last incus start, otherwise cloud-init or similar may re-trigger and potentially make a bit of a mess.

5

I’ve tried the process with one of my containers and it worked perfectly fine! Thank you so much for pointing me to the “–refresh” flag :slight_smile: