Abstract
Allow for users to provide a Starlark scriptlet that decides on cluster target at instance creation time.
This scriptlet would be provided with information about the new requested instance as well as a list of candidate cluster members (online and compatible with the request) and details about them and their existing instances to make the decision.
It will have the ability to specify where the instance will be created or can prevent the instance from being created at all.
Rationale
This allows for custom logic to control instance placement, rather than the very basic placement logic that LXD currently has (cluster member with fewest instances).
Specification
Design
The instance placement scriptlet would be provided to LXD by way of a global configuration option called instances.placement.scriptlet. This would be stored in the global database and available to all cluster members.
When a request for a new instance comes to POST /1.0/instances without a target URL parameter specified the the instance placement scriptlet (if defined) would be executed on the cluster member that the request arrived at.
The scriptlet environment will be able to access the following info:
- Instance create request (including expanded config and devices from profiles).
- Profiles due to be used.
- Reason for instance request:
- New instance request.
- Temporary instance migration during evacuation.
- Temporary instance relocation migration while a cluster member is marked as dead.
- Ability to retrieve candidate cluster members and their config. Cluster member candidates will be filtered by:
- Online.
- Correct architecture (by resolving image arch).
- Available to restricted project (member groups).
- Ability to retrieve cluster memberâs state metrics (including system load, storage pool state etc).
- Ability to retrieve cluster memberâs resources info.
- Ability to retrieve the expected resources required for the instance.
API changes
A new server config key called instances.placement.scriptlet will be added along with an API extension called instances_placement_scriptlet.
There will be new struct types added:
// InstancePlacement represents the instance placement request.
//
// API extension: instances_placement_scriptlet.
type InstancePlacement struct {
api.InstancesPost `yaml:",inline"`
Reason string `json:"reason"`
Project string `json:"project"`
}
// InstanceResources represents the required resources for an instance.
//
// swagger:model
//
// API extension: instances_placement_scriptlet.
type InstanceResources struct {
CPUCores uint64 `json:"cpu_cores"`
MemorySize uint64 `json:"memory_size"`
RootDiskSize uint64 `json:"root_disk_size"`
}
Scriptlet definition
The instance placement scriptlet will be expected to contain a function called instance_placement.
This function will be called by LXD when an instance is to be created, and will be provided with the instance creation request (equivalent to the InstancePlacement struct above) as the request argument to the function. This will include a Reason field that will indicate the reason for the request, which can be new, evacuation or relocation.
As Starlark has no concept of exceptions, instead the instance_placement function can return an error by returning a non-none value. This value will be returned as an error response to the caller of LXDâs REST API. This allows the scriptlet to block the creation of the instance if needed.
The scriptlet can also optionally control which cluster member the instance should be created on.
To do this it can call the set_target function with the member_name parameter indicating the cluster member it wants.
Functions that will be available to the scriptlet:
log_info(*messages): Add a log entry to LXDâs log at info level.messagesis one or more message arguments.
log_warn(*messages): Add a log entry to LXDâs log at warn level.messagesis one or more message arguments.
log_error(*messages): Add a log entry to LXDâs log at error level.messagesis one or more message arguments.
set_target(member_name): Set the cluster member where the instance should be created.member_nameis the name of the cluster member the instance should be created on.
get_cluster_member_state(member_name): Get the cluster memberâs state. Returns an object with the cluster memberâs state equivalent toapi.ClusterMemberState.member_nameis the name of the cluster member to get state for.
get_cluster_member_resources(member_name): Get information about resources on the cluster member. Returns an object with the resource info equivalent toapi.Resources.member_nameis the name of the cluster member to get resource info for.
get_instance_resources(): Get information about the resources the instance will require. Returns an object with the resources info equivalent toapi.InstanceResources.
The scriptlet must implement:
instance_placement(request, candidate_members)requestwill be an object containing an expanded representation ofshared/api/scriptlet.InstancePlacement.candidate_memberswill be a list of cluster member objects representingshared/api.ClusterMemberentries.
Example implementation:
def instance_placement(request, candidate_members):
log_info("instance_placement started: ", request)
if request.name == "foo":
log_error("Invalid name supplied: ", request.name)
return "Invalid name"
set_target(candidate_members[0].server_name)
return # No error
Storing a scriptlet in LXD can be achieved by creating a file for the scriptlet, e.g. instancePlacement.star and then using the following command:
lxc config set instances.placement.scriptlet "$(cat instancePlacement.star)"
Example of a scriptlet error returned to the caller:
lxc init images:ubuntu/jammy foo
Creating foo
Error: Failed instance creation: Failed instance placement scriptlet: Failed with return value: "Invalid name"
CLI changes
No CLI changes are expected.
Database changes
No DB changes are expected.
Upgrade handling
As this is a new feature, no upgrade handling is required.
Further information
When the instances.placement.scriptlet setting is changed, the new value will be compiled to check that it is a valid Starlark program. If successful then a cached compiled program will be held in memory ready to be run for each new instance creation request to avoid having to load it from the database and compile it every time.
Because each cluster member requires to have its own in-memory compiled cache of the program, when the instances.placement.scriptlet setting is changed we will rely upon the existing mechanism that notifies the other cluster members to refresh their local compiled cache of the programâŠ