I’m looking to build an Inucs lab tending more towards mimicking production than not. For example:
if implementing OVN, it will be with certs
if implementing ceph, there will be dedicated storage networking
OIDC for WebUI, if using the UI is important
and etc. (my examples might be bad ones–feel free to let me know)
Today, my question is about drives.
If you were setting up a hardware template for lab, what would storage look like? If you were training people for Incus admin jobs, would you have an ideal drive setup?
If all you can provide is “it depends on what you’re going to do,” I get it.
Let’s assume the lab is going to be a 3+ node cluster with ceph. At a minimum, I’m planning to have a boot drive and a ceph drive. With the equipment at hand, I have room for 2 (possibly 3) more drives. Would it be smart to have a mirrored 2-drive ZFS setup in addition to the ceph drive? What would go there, if ceph is already available for storage?
Also, how small can the Incus boot drive be? I believe I have a Debian-based freshly-installed instance where only 6 GiB appears to be used for now. If the boot drive drive is slow, what will the impact be on performance, assuming containers and VMs are on fast ceph or ZFS storage?
Ceph can be fast but it tends to be pretty high latency so it’s often useful to also have some local storage available for workloads that are more latency sensitive or that don’t need the redundancy.
For my own clusters I do tend to go with something along the lines of:
Boot drive (NVME)
50GiB root partition
Rest used as mix of ZFS cache and storage of Ceph journals for non-NVME disks
2x NVME SSDs for local storage
Anything else I can fit in the machine goes into Ceph
I generally consider the boot drive to be disposable, the OS can be very easily reinstalled and no data of consequence is stored on there. That said, I tend to go with pretty good quality NVME drives so I’m yet to have any of those die yet
The choice of NVME above for the local storage is also mostly because of practicality in my case.
I tend to use regular M.2 NVME drives for those two and put them on a PCIe adapter that can use some of my unused PCI slots.
Then all the nice hotswap bays get used for the Ceph disks with a mix of SSD and HDD depending on use cases. Note that this all makes sense because most of my servers predate U.2 compatible backplanes. If you have a backplane that can do NVME U.2 / 2.5" SATA / 2.5" SAS, then you have even more flexibility and may be able to use the PCI slots for faster networking or GPUs.
On the performance front, be careful with the Incus database. It doesn’t need high throughput, but high latency on it can become noticeable as in a cluster, every database write turns into 3 to 5 writes (one for each DB voter + one for each DB standby if you have more than 3 servers).
As far as disk usage on the root disk. Very little gets stored there if you make sure to create volumes on your local pool and set storage.backups_volume and storage.images_volume accordingly. At that point the only thing that Incus will really be writing to will be the database and random config and log files.
Depending on what you use local storage for that may be overkill though.
It’s pretty rare for good quality NVME to die and as local storage is usually used for less critical things (as the server itself may still go down), you may opt for capacity over reliability.
