I’ve been playing with IncusOS on a spare node, and wanted to start thinking about installing it on my main host: I’ve currently got Ubuntu 24.04 Server (minimal) installed only to run Incus (stable) and my only concern is the maintenance of this OS (and having to provide remote access to achieve this). IncusOS addresses this for me.
The host has an NVMe drive for the OS, a quad NVMe configured as a ZFS storage pool for containers, and a quad SATA configured as a ZFS storage pool (as yet unused).
I can see that I can configure the installation target using the image customiser, but will installing IncusOS leave the two pools alone? Can I then attach them and recover the containers? I appreciate that I might need to backup the containers from the NVMe pool to the SATA pool as part of this. I’m able to move the containers off the host during the migration, but was wondering how much I can actually do in place.
IncusOS will only wipe the boot drive during installation, so your other pools would remain intact.
But I don’t think that IncusOS supports importing unencrypted storage pools post installation (@gibmat can correct me there).
We’d need to check if incus admin recover can be used over the network (it uses an internal API which may need tweaking to be allowed) and if that would then be sufficient to temporarily import a pool as you re-shuffle things around.
Basically a migration like this would then look like:
Move all instances from NVME to SATA pool
Reinstall the OS (and make sure to fetch the storage recovery key)
Wipe the NVME disks
Create a new NVME pool through IncusOS
Ads new NVME pool to Incus
Use recover to get the SATA pool back
Move all instances to NVME
Delete SATA pool from Incus
Wipe the SATA disks
Create a new SATAV pool through IncusOS
Ads new SATA pool to Incus
Move instances back onto SATA
That will then get you 3 Incus pools:
local (remaining space on boot disk, useful for images and backup prep space)
nvme
sata
As all pools will have been recreated, they’ll all be properly encrypted with their keys stored on the encrypted root partition which is itself encrypted through the TPM, giving you a proper encryption chain that can bring everything up automatically on boot.
As mentioned, we’ll want to test the recover logic to make sure that those steps will indeed work.
The alternative would be to use storage on another temporary machine to hold incus export --optimized for each instance and volume, then the everything on your server can be wiped clean before importing the instances back from a back up.
@gibmat could you test the plan above in a VM? If it works we should add a tutorial section to the documentation and have an entry for this scenario. If it doesn’t, we’ll need an issue against Incus to review the handling of the internal API so we can have recover be used on IncusOS (same with sql and some of those other commands).
Yeah, we will need to do some tweaking of Incus so we can run recover against a remote server. Other than that a migration like you outlined works quite smoothly and I’ve got a basic tutorial sketched out.
