Cluster scheduler places a VM on a non-KVM member, only failing after an existing instance is already destroyed

1

On a 3-member cluster where 2 members lack /dev/kvm, creating a virtual-machine-type instance with no explicit target can land on a non-KVM member. The instance record creation then fails.

Failed creating instance record: Instance type “virtual-machine” is not supported on this server: KVM support is missing (no /dev/kvm)

This bit me specifically via Terraform’s replace flow: it destroys the existing instance first, then tries to create the replacement, which failed twice in a row landing on the same non-KVM member (not obviously random).

I worked around it with incus cluster set scheduler.instance=manual on the non-KVM members, which seems to be the intended mechanism for this.

Before filing a GitHub issue: is capability-aware scheduling (skipping members that can’t run the requested instance type) something the scheduler is expected to handle automatically, or is scheduler.instance=manual / cluster groups the documented/expected way to handle mixed-capability clusters? If the latter, is there a doc page I missed that should have pointed me there sooner?

2

The easiest way to handle this is to create a cluster group, say vm, put the servers that are VM capable in that group and then create instances with --target @vm.

You could also load up a placement scriptlet that would then skip that specific server for virtual-machine instances.

Or you can indeed us scheduler.instance=manual so that server doesn’t get any instance created on it automatically and requires an explicit --target to get an instance on there.

Our built-in scheduler only places things to try and even out the instance count across all servers. It doesn’t care about instance type, system capabilities, available system resources or instance sizes. But we do make all that data available to the placement scriptlet so you can adjust things to match your environment.