IncusOS and Ceph Cluster with Single Drives?

IncusOS
1

I’m running a single Incus 6 server on Ubuntu Server and when running properly(99% of the time), Incus has made my life much easier. When issues occur, 90+% of them have been storage-related.

I have a few mini PCs that can only hold a single 1TB or 2TB drive each. I wanted to turn them into an IncusOS and Ceph cluster, but I’m new to IncusOS, clustering, and Ceph. Each device is supposed to have IncusOS, the default Incus Storage Pool (for instances), and then either one or two Ceph OSD partition(s) (possibly implemented as an Incus Storage Pool that only the Ceph OSD container uses).

Architectural Rationale:

  • When running Incus 6, I’ve frequently had instances go significantly over their quota. On a few occasions, the entire storage pool filled, all instances writing to the storage pool crashed, and the storage pool had to be manually cleaned.
  • I’d prefer to keep Ceph and the default Incus Storage Pool on separate partitions to eliminate any interactions.
  • Plus, from what I’ve read, Ceph prefers exclusive access to the partitions.
  • Ceph only uses the smallest partition size available, so with a mix of 1TB and 2TB drives, using uniform 800GB partition sizes would mean that the Incus Storage Pools would be ~150GB(1TB) or ~350GB(2TB), which is significantly larger than required for my workloads.

However, when I started trying to build the cluster, I found out:

  • I cannot delete the automatically created ZFS partition,

  • the automatically created ZFS partition that uses the remainder of the drive is unable to shrink - but can expand.

  • Even with the flasher-tool, there does not appear to be a way to modify the partitioning scheme or to tell the installer to create 2-3 ZFS partitions, one (or two) size == 800GB, and the other that uses the remainder of the drive.

So, I’ve got this hardware that does not seem to be capable of doing what I intended, but I’d really prefer to use IncusOS instead of Ubuntu Server, so I’m hoping I’ve missed something or I’m misunderstanding something.

Is there a way to get IncusOS to create a modified partition scheme on install? Or is there a way to divide the automatically created ZFS partition?

Is my architecture inherently broken and needs to be modified or something? Or, is it invalid on an immutable OS for some reason??

Is there a complete yaml schema for flasher-tool and IncusOS documented somewhere? If not, is there a recommended place to add it to the documentation??

What other simple tweaks would allow building the described cluster on IncusOS?

Thanks for everyone’s efforts toward making IncusOS so amazing.

2

TL;DR:

  1. Is it even possible for a Ceph OSD run in a container on IncusOS while having exclusive access to a defined portion of the ONLY physical drive in the device?
  2. Can I create an additional partition or two before the automatically created “remainder” ZFS partition while installing IncusOS?
  3. Is there a complete yaml schema for flasher-tool and IncusOS documented somewhere? If not, is there a recommended place to add it to the documentation??
3

No, the partitioning scheme is fixed. IncusOS uses systemd-repart to create partitions (definitions here, some reference documentation is also available here), and there’s no (official) way to influence this since the configuration files are baked into the IncusOS image. We don’t have any way to inject changes during the initrd phase of startup, which is when systemd-repart does its work.

I wouldn’t say broken, but maybe not quite the best fit for running an IncusOS-based cluster. IncusOS assumes it controls the disk it is installed to/running from, which allows the system to be fully operational out-of-the-box. This works great for stand-alone systems, and in clustered environments it’s assumed additional storage pools will be backed by other independent devices.

The seeds are defined by the structs in the api/seed package. We don’t currently have any nicely rendered documentation generated from them.

:warning: Caution, here be dragons :warning:

This is entirely untested, and will put you well outside of normal support, but I think it could work.

When the IncusOS install process completes, the target disk will contain 8 partitions. On first boot, IncusOS then creates the encrypted swap, root, and zfs partitions. If you don’t allow the system to reboot post-install, but rather boot a live Linux distro you could create an additional partition for use by Ceph. Then, reboot and let IncusOS perform its first-boot actions. (You’ll need to disable, then enable Secure Boot as part of this process, but don’t wipe any of the installed certificates.)

I think this should work, as IncusOS depends on partition codes and labels to mount things. It should essentially ignore your extra partition, which you could then pass through into a Ceph container.

Important considerations:

  • You must leave at least 35GiB of free space for IncusOS to create the final three partitions on its first boot
  • Your extra partition won’t be encrypted like the rest of the IncusOS-created storage
  • IncusOS won’t be able to manage or report about the extra partition
  • Again, you’d be well out of support if something does go wrong
4

@gibmat:
Thanks for your wonderfully meticulous reply. It really helped clarify the current situation, but I think there’s more complexity just ahead.

First, your reply seems to confirm my understanding from the Ceph documentation that the Ceph OSD container requires direct exclusive access to the storage block device, and cannot use any part of the Incus storage pool. Ok, at least I’m not skipping over a more obvious solution.

Secondly, that got me thinking about the next steps, and how even if I manage to fight back the dragons and create the additional partition, can a Ceph OSD container even see the storage that it needs to access?? And given that the Ceph instance I’m trying out on IncusOS 7.1 only has options for “Attach custom volume” or “Mount host path”, I assume that somehow or another, I still have to get IncusOS to see that partition so that it can pass it to the Ceph OSD container.

I know you’ve been working on Add ability to automatically deploy a Ceph cluster and add it as an Incus storage pool #1159, and there’s a point there where you state:

Each ODS instance utilizes a LUKS-encrypted raw storage device from the IncusOS server on which it is running.

So, how do I turn the extra partition into a raw storage device that IncusOS can provide to the Ceph instance??

Reading back through all the documentation, I’m noticing that IncusOS might ONLY be able to interact with entire drives. For example, the Preparing a storage volume for Incus - IncusOS documentation tutorial states that “This tutorial will use a single drive for simplicity.”, but looking at the Storage API, it also only contains commands and examples for entire disks, where the non-storage volume sections like “wiping a drive” and Raw Drive Encryption only refer to entire drives.

And then I found Using encrypted disks in IncusOS - #7 by stgraber where Stéphane confirms:

Yeah, we’ve got it restricted to entries from /dev/disk/by-id/ and then further check that the disk isn’t a member of a pool or the boot drive.

So, ultimately, it seems like even if I can create the extra partition, I still need IncusOS to see it to pass it to the Ceph OSD container, and IncusOS is restricted from seeing the extra partition, so I think I’m stuck. Or I’m misunderstanding something, which is a very real possibility after banging my head against this problem.

And the weird part of all this – that is really making me feel like I’m misunderstanding some tiny, yet crucial point – is that IncusOS can be run virtually on Incus or IncusOS. And if nested IncusOS deployments are valid, then IncusOS presumably has the ability to take the virtualized storage it is running on and provide it to instances, but that is likely not raw enough for Ceph OSD - for good reason.

Is this impossible with Incus 7.1? Or am I misunderstanding something?

5

OSDs do need a raw block device; the easiest and most performant way is using a physical disk. It would also be possible to use Incus to create a custom block device that could then be attached to a Ceph virtual machine (not supported by containers). The device would be stored as a zvol in the Incus zfs pool. I don’t know what sort of performance penalty you’d be looking at, but that could be a possible path forward.

Creating the Ceph container would be done with the normal Incus API, so if you knew that (for example) your custom partition was /dev/sda9, you’d just use that in your creation command. IncusOS would recognize the partition table on /dev/sda, but that would be the extent of what the OS does for you.

Yes, we’re working on a way to automate deployment of a Ceph cluster. :smile: Part of that work is taking a drive, auto-encrypting it, then passing that LUKS-backed device for use by the OSD. As I mentioned above, there’s nothing that would stop you from manually using a specific partition when you create your container.

Yes, IncusOS assumes it will be working with full drives, not a partition of a drive.

When running IncusOS as a guest (or any other VM for that matter), the IncusOS host will create zvol(s) from its zfs pool as mentioned above. You can nest down a few levels of VMs if you really want, but performance does start to drop of at some point due to compounding levels of virtualization and redirection.

6

I have a few mini PCs…

Unless things with CEPH have changed in the past 6 months, unless “a few” means 15+ and those drive are enterprise drives that don’t “get a bit flaky sometimes because they were used a bit too much”, you are probably setting yourself up for a lot of pain. If it’s a homelab and you want to experiment, it’s all good but given that your motivation seems to be stemming from painful failures, it’s unlikely CEPH is going to improve the situation here. But if you want to see by yourself (and maybe finding out I was wrong), don’t let this stop you. :slight_smile:

7

I’ve been running “small” Ceph clusters for a little while, basically 3 servers, 4 drives each. But those are indeed pretty expensive enterprise SATA SSD (older servers, no U.2 on those).

I have one cluster on “good” consumer SSDs (Samsung EVO Pro series) and the difference compared to the enterprise drives is quite noticeable even though they’re in much faster servers…

That said, it’s all usable, it’s just a matter of how much latency do you perceive. Overall throughput on very large files is usually okay.