@gibmat:
Thanks for your wonderfully meticulous reply. It really helped clarify the current situation, but I think there’s more complexity just ahead.
First, your reply seems to confirm my understanding from the Ceph documentation that the Ceph OSD container requires direct exclusive access to the storage block device, and cannot use any part of the Incus storage pool. Ok, at least I’m not skipping over a more obvious solution.
Secondly, that got me thinking about the next steps, and how even if I manage to fight back the dragons and create the additional partition, can a Ceph OSD container even see the storage that it needs to access?? And given that the Ceph instance I’m trying out on IncusOS 7.1 only has options for “Attach custom volume” or “Mount host path”, I assume that somehow or another, I still have to get IncusOS to see that partition so that it can pass it to the Ceph OSD container.
I know you’ve been working on Add ability to automatically deploy a Ceph cluster and add it as an Incus storage pool #1159, and there’s a point there where you state:
Each ODS instance utilizes a LUKS-encrypted raw storage device from the IncusOS server on which it is running.
So, how do I turn the extra partition into a raw storage device that IncusOS can provide to the Ceph instance??
Reading back through all the documentation, I’m noticing that IncusOS might ONLY be able to interact with entire drives. For example, the Preparing a storage volume for Incus - IncusOS documentation tutorial states that “This tutorial will use a single drive for simplicity.”, but looking at the Storage API, it also only contains commands and examples for entire disks, where the non-storage volume sections like “wiping a drive” and Raw Drive Encryption only refer to entire drives.
And then I found Using encrypted disks in IncusOS - #7 by stgraber where Stéphane confirms:
Yeah, we’ve got it restricted to entries from /dev/disk/by-id/ and then further check that the disk isn’t a member of a pool or the boot drive.
So, ultimately, it seems like even if I can create the extra partition, I still need IncusOS to see it to pass it to the Ceph OSD container, and IncusOS is restricted from seeing the extra partition, so I think I’m stuck. Or I’m misunderstanding something, which is a very real possibility after banging my head against this problem.
And the weird part of all this – that is really making me feel like I’m misunderstanding some tiny, yet crucial point – is that IncusOS can be run virtually on Incus or IncusOS. And if nested IncusOS deployments are valid, then IncusOS presumably has the ability to take the virtualized storage it is running on and provide it to instances, but that is likely not raw enough for Ceph OSD - for good reason.
Is this impossible with Incus 7.1? Or am I misunderstanding something?