My use case is similar to what I have seen @stgraber do on a few occasions. To have a few incus nodes in a cluster which are not bare metal machines, but actually VMs running on other nodes. This is not an issue, as these VMs can be IncusOS installations themselves. The issue with my use case is that I need to use a thin layer of containers instead of VMs.
More detailed use case
I’m a university professor managing a dual use incus cluster. The first usage is to run research optimisation experiments which need dedicated resources on a controlled environment. The second use case is to study running services for our research group. Due to (very) limited resources, we can’t have dedicated machines for doing one or the other. Furthermore, it’s also an interesting research experience.
Projects are not great for separating the usage limits as they only care about the total usage, not a limit per machine. One solution is to use an instance scheduler and default profiles with a few projects. The scheduler checks total machine allocated RAM and configured CPU cores, but the scheduler is not used on instance configuration change, so the limits are not enforced then. The best solution I’ve found so far is to have cluster nodes which are containers which are used just to limit their CPU core usage and RAM and use them in cluster groups for resource segregation. One cluster group for optimisation and another for running services.
VMs use too many resources on our very low resource pool and they are also a bit less controlled for running the optimisation experiments with the highest resource usage possible. Many experiments already take quite a few months to run. I got other professors to accept this solution as long as the resource expenditure is minimal.
Possible solutions
- One possible solution would be to have a mixed cluster with IncusOS and other nodes we run right now for resource separation. But there needs to be a way to synchronise an upgrade between the IncusOS nodes with the others. But this also defeats a little bit of the main advantage of IncusOS to control the incus nodes.
- Another, interesting solution would be to have IncusOS container images. They have the same administration control as the bare metal/VM nodes, but run in containers. They are not as controlled in having secure boot together with a TPM, but would help quite a lot.
My guess is that the first one might be too complicated for a problem which might not even be an ideal solution. For the second one, incus containers have been shown to reproduce many different linux distributions/environments, so I guess IncusOS containers are possible. The main issue would be similar to running IncusOS without SecureBoot or TPM. Since incus follows the model of security by design, it might not be a desirable solution.
Would any of these two solutions be interesting for IncusOS in the long run? Or does it not make sense even to think about them?