When setting project limits on a cluster, say for example limits.memory=20GiB, this limit applies to all instances inside the project. What I really want is for this limit to apply per cluster member inside the project.
Since I have 3 servers restricted for this project and each should be limited to 20GB, I set the limit to 60GB, but it still doesn’t seem proper since if I launch three instances with 10GB of memory and then one with 20GB, one server will get two instances with total of 30GB of memory.
Is it possible to set project limits per server instead of being for the entire project? It’s also interesting that I’m getting a fourth server up soon and each time I get a new one, I need to change project configurations while it the limit was set per server, no change would be needed.
Although that would work on limits, I think that’s too much of a hassle to consider right now. First I’ll study what’s implemented on the starlark script for instance allocation. First because the best allocation strategy should probably be to schedule an instance into a server which leaves the least leftover resources (considering only memory, for example). And second because checking leftover resources for users is quite complicated. They would have to go over a few projects and check configs on instances and compare to limits. That’s not nice for them.
Is the current implementation of the scheduling script able to do a least leftover resources allocation? If so, where should I study how to write one of these scripts?