Placement scriptlet issues on Incus 6.14

My instance placement scriptlet stopped working after the upgrade to incus 6.14.

Currently, even replacing the scriptlet to debug it is not working. I’m using the same format provided in the scriptlet incus documentation.

pargo@bastion:~/scriptlet$ cat maintenance.py 
# Error return strings
error_maintenance           = "Em manutenção."

# Main instance placement scriptlet
def instance_placement(request, candidate_members):
    log_error("SCRIPTLET DEBUG Request: profiles = ", request.profiles, ", name = ", request.name, ", image alias = ", request.source.alias, ", reason = ", request.reason, ", project = ", request.project, "\nSCRIPTLET DEBUG Candidade members: ", [x.server_name for x in candidate_members], "\nSCRIPTLET DEBUG Request config: ", request.config)
    project = get_project( request.project )
    log_error("SCRIPTLET DEBUG Request: ", request, "\nSCRIPTLET DEBUG Candidade members: ", candidate_members, "\nSCRIPTLET DEBUG Project: ", project)
    fail(error_maintenance)
    return          # No limits to check, so scheduling is left for the default scheduler

pargo@bastion:~/scriptlet$ cat maintenance.py | incus config set instances.placement.scriptlet=-

Above I set the new maintenance placement scriptlet. Then I check for the placement scriptlet and it’s the old one (which stopped working for some unknown reason).

pargo@bastion:~/scriptlet$ incus config get instances.placement.scriptlet                                                                                                                                                                    
# Error return strings                                                                                                                                                                                                                       
error_cpu_not_string        = "Configuração de limits de CPU não é string."                                                                                                                                                                  
error_cpu_syntax            = "Sintaxe inválida de limite de CPU."                                                                                                                                                                           
error_cpu_not_pinned        = "Núcleos de CPU não foram fixados."                                                                                                                                                                            
error_cpu_key               = "Chave limits.cpu com erro: "                                                                                                                                                                                  
error_pinned_cpu_not_allowed = "CPUs fixados pela instância não são permitidos pelo projeto."                                                                                                                                                
                                                                                                                                                                                                                                             
error_mem_not_string        = "Configuração de limite de memória não é string."                                                                                                                                                              
error_mem_syntax            = "Sintaxe inválida de limite de memória. "                                                                                                                                                                      
error_mem_key               = "Chave limits.memory com erro: "                                                                                                                                                                               
error_mem_not_fixed         = "Memória não fixada."                                                                                                                                                                                          
error_mem_limit_exceeded    = "Memória acima do permitido pelo projeto."                                                                                                                                                                     
                                                                                                                                                                                                                                             
error_responsible_undefined = "Responsável não definido para instância na chave user.responsavel."                                                                                                                                           
error_project_config        = "Erro na configuração do projeto. Por favor fale com um administrador do cluster para resolver o problema. Erro: "                                                                                             
error_no_available_node     = "Não há máquina válida disponível."                                                                                                                                                                            
                                                                                                                                                                                                                                             
# Parses limits.cpu type strings which should have pinned cpus. Returns either an error string or a tuple containing the cpu bit vector and the most significant set bit                                                                     
def parse_cpu_pin(cpu_string):                                                                                                                                                                                                               
    cpu_bit_vector  = 0                                                                                                                                                                                                                      
    mssb            = -1                                                                                                                                                                                                                     
                                                                                                                                                                                                                                             
    if type(cpu_string) != "string":                                                                                                                                                                                                         
        return error_cpu_not_string                                                                                                                                                                                                          
    if cpu_string.isdigit():                                                                                                                                                                                                                 
        return error_cpu_not_pinned                                                                                                                                                                                                          
    parts = cpu_string.split(",")                                                                                                                                                                                                            
    for p in parts:                                                                                                                                                                                                                          
        r = p.split("-")                                                                                                                                                                                                                     
        if len(r) not in (1, 2):                                                                                                                                                                                                             
            return error_cpu_syntax                                                                                                                                                                                                          
        if len(r) == 1:
            if not r[0].isdigit():
                return error_cpu_syntax
            cpu = int(r[0])
            cpu_bit_vector |= 1 << cpu
            if cpu > mssb:
                mssb = cpu
        else:
            if (not r[0].isdigit()) or (not r[1].isdigit()):
                return error_cpu_syntax
            min = int(r[0])
            max = int(r[1])
            if max < min:
                return error_cpu_syntax
            for cpu in range(min, max+1):
                cpu_bit_vector |= 1 << cpu
            if max > mssb:
                mssb = max
    return cpu_bit_vector, mssb

# Parses limits.memory type strings and returns the memory size in bytes or an error string
def parse_memory(memory_string):
    binary_suffixes     = ("KiB", "MiB", "GiB", "TiB", "PiB", "EiB")
    decimal_suffixes    = ( "kB",  "MB",  "GB",  "TB",  "PB",  "EB")

    if type(memory_string) != "string":
        return error_mem_not_string
    if memory_string.endswith(binary_suffixes):
        if not memory_string[:-3].isdigit():
            return error_mem_syntax 
        mem = int(memory_string[:-3])
        for bs in binary_suffixes:
            mem <<= 10
            if memory_string.endswith(bs):
                break
    elif memory_string.endswith(decimal_suffixes):
        if not memory_string[:-2].isdigit():
            return error_mem_syntax 
        mem = int(memory_string[:-2])
        for ds in decimal_suffixes:
            mem *= 1000
            if memory_string.endswith(ds):
                break
    elif memory_string.endswith("B"):
        if not memory_string[:-1].isdigit():
            return error_mem_syntax 
        mem = int(memory_string[:-1])
    else:
        return error_mem_syntax 
    return mem

# Parses limits.memory type strings and returns the memory size in bytes or an error string
def parse_memory(memory_string):
    binary_suffixes     = ("KiB", "MiB", "GiB", "TiB", "PiB", "EiB")
    decimal_suffixes    = ( "kB",  "MB",  "GB",  "TB",  "PB",  "EB")

    if type(memory_string) != "string":
        return error_mem_not_string
    if memory_string.endswith(binary_suffixes):
        if not memory_string[:-3].isdigit():
            return error_mem_syntax 
        mem = int(memory_string[:-3])
        for bs in binary_suffixes:
            mem <<= 10
            if memory_string.endswith(bs):
                break
    elif memory_string.endswith(decimal_suffixes):
        if not memory_string[:-2].isdigit():
            return error_mem_syntax 
        mem = int(memory_string[:-2])
        for ds in decimal_suffixes:
            mem *= 1000
            if memory_string.endswith(ds):
                break
    elif memory_string.endswith("B"):
        if not memory_string[:-1].isdigit():
            return error_mem_syntax 
        mem = int(memory_string[:-1])
    else:
        return error_mem_syntax 
    return mem

# Counts used threads in a cpu_bit_vector up to bit number mssb
def count_used_threads(cpu_bit_vector, mssb):
    used_threads    = 0
    for i in range(1+mssb):
        if cpu_bit_vector & (1 << i) != 0:
            used_threads += 1
    return used_threads

# Main instance placement scriptlet
def instance_placement(request, candidate_members):
    #log_error("SCRIPTLET DEBUG Request: profiles = ", request.profiles, ", name = ", request.name, ", image alias = ", request.source.alias, ", reason = ", request.reason, ", project = ", request.project, "\nSCRIPTLET DEBUG Candidade me
mbers: ", [x.server_name for x in candidate_members], "\nSCRIPTLET DEBUG Request config: ", request.config)
    project = get_project( request.project )
    # log_error("SCRIPTLET DEBUG Request: ", request, "\nSCRIPTLET DEBUG Candidade members: ", candidate_members, "\nSCRIPTLET DEBUG Project: ", project)
    # Check for user.node.represented project key
    rep_unique  = False
    if "user.node.represented" in project.config:
        if project.config["user.node.represented"] == "true" and "user.responsavel" not in request.config:
            fail(error_responsible_undefined)
        responsavel = request.config["user.responsavel"]
        if "user.node.represented.unique" in project.config:
            if project.config["user.node.represented.unique"] == "true":
                rep_unique = True
    # Check for user.node.limits.cpu project key
    cpu_pin = False
    if "user.node.limits.cpu" in project.config:
        cpu_pin = True
        ret = parse_cpu_pin(project.config["user.node.limits.cpu"])
        if type(ret) == "string":
            fail(error_project_config, ret)
        project_cpu_bit_vector  = ret[0]
    # Check for user.node.limits.cpu.unique project key
    cpu_unique = False
    if "user.node.limits.cpu.unique" in project.config:
        if project.config["user.node.limits.cpu.unique"] == "true":
            cpu_unique = True

    if cpu_pin or cpu_unique:
        if "limits.cpu" not in request.config:
            fail(error_cpu_not_pinned)
        ret = parse_cpu_pin(request.config["limits.cpu"])
        if type(ret) == "string":
            fail(error_cpu_key, ret)
        instance_cpu_bit_vector = ret[0]
        if cpu_pin and (instance_cpu_bit_vector | project_cpu_bit_vector) != project_cpu_bit_vector:
            fail(error_pinned_cpu_not_allowed)
    # Check for user.node.limits.memory project key
    mem_limited = False
    if "user.node.limits.memory" in project.config:
        mem_limited = True
        ret = parse_memory(project.config["user.node.limits.memory"])
        if type(ret) == "string":
            fail(error_project_config, ret)
        req_mem_limit = ret
        if "limits.memory" not in request.config:
            fail(error_mem_not_fixed)
        ret = parse_memory(request.config["limits.memory"]) 
        if type(ret) == "string":
            fail(error_mem_key, ret)
        req_mem_limit -= ret
        if req_mem_limit < 0:
            fail(error_mem_limit_exceeded)

    if rep_unique or mem_limited or cpu_unique:
        machines = dict()
        # Register initial values for relevant statistics from candidate members
        for m in candidate_members:
            mvars = dict()
            mvars["valid"] = True
            if mem_limited:
                mvars["req_mem"]    = req_mem_limit
            if cpu_unique:
                mvars["used_cpu"]   = 0
                mvars["mssb"]       = -1
            machines[m.server_name] = mvars
        # Get valid machines and relevant statistics
        inst = get_instances(project=request.project)
        for i in inst:
            if i.location in machines and machines[i.location]["valid"]:
                if rep_unique:
                    if "user.responsavel" not in i.expanded_config or i.expanded_config["user.responsavel"] != responsavel:
                        machines[i.location]["valid"] = False
                        continue
                if mem_limited:
                    if "limits.memory" not in i.expanded_config:
                        machines[i.location]["valid"] = False
                        continue
                    ret = parse_memory(i.expanded_config["limits.memory"])
                    if type(ret) == "string":
                        machines[i.location]["valid"] = False
                        continue
                    machines[i.location]["req_mem"] -= ret
                    if machines[i.location]["req_mem"] < 0: 
                        machines[i.location]["valid"] = False
                        continue
                if cpu_unique:
                     if "limits.cpu" not in i.expanded_config:
                        machines[i.location]["valid"] = False
                        continue
                    ret = parse_cpu_pin(i.expanded_config["limits.cpu"])
                    if type(ret) == "string":
                        machines[i.location]["valid"] = False
                        continue
                    machines[i.location]["used_cpu"] |= ret[0]
                    if instance_cpu_bit_vector & machines[i.location]["used_cpu"] != 0:
                        machines[i.location]["valid"] = False
                        continue
                    if machines[i.location]["mssb"] < ret[1]:
                        machines[i.location]["mssb"] = ret[1]
        # Choose target node
        target = None
        for mname in machines:
            if machines[mname]["valid"]:
                if cpu_unique:
                    used_threads = count_used_threads(machines[mname]["used_cpu"], machines[mname]["mssb"])
                better_target   = False
                if target == None:
                    better_target = True
                elif mem_limited and machines[mname]["req_mem"] < machines[target]["req_mem"]: # prefer best fit into memory
                    better_target = True
                elif cpu_unique and used_threads > target_used_threads: # prefer best fit on cpu cores
                    better_target = True
                if better_target:
                    target = mname
                    if cpu_unique:
                        target_used_threads = used_threads  
        if target == None:
            fail(error_no_available_node)
        set_target(target)
    return          # No limits to check, so scheduling is left for the default scheduler
pargo@bastion:~/scriptlet$ 

I’m having issues to set any server configuration keys at the moment actually. Not sure what happened.

pargo@bastion:~/scriptlet$ incus config set user.foo=bar
pargo@bastion:~/scriptlet$ incus config get user.foo

pargo@bastion:~/scriptlet$ incus config set user.foo="bar"
pargo@bastion:~/scriptlet$ incus config get user.foo

That’s pretty odd. No error when setting it?

I usually would use cat foo.py | incus config set instances.placement.scriptlet - rather than the =- syntax, but the parser probably see them as the same thing so unlikely to be related.

Might be worth running incus monitor --pretty while running the config update to see if something useful gets logged.

The monitor is not showing anything. I currently can change instance keys, but not server keys, so I really don’t know what’s up.

Same behavior if you try to do incus config set user.foo=bar?

Yep. Same behaviour.

pargo@bastion:~/scriptlet$ incus config set user.foo=bar
pargo@bastion:~/scriptlet$ incus config get user.foo

pargo@bastion:~/scriptlet$ incus config set user.foo="bar"
pargo@bastion:~/scriptlet$ incus config get user.foo

Nothing shows up in monitor as well. Tried restarting most cluster machines (the ones not running critical stuff at least).

Not having any luck reproducing this here. What’s very odd is the lack of any error as if the change makes it all the way but doesn’t persist in the DB somehow.

Maybe check incus admin sql global "SELECT * FROM config" to see what’s actually in the DB.

Just figured something out. This error is happening only while trying to control the cluster from a remote server. From inside an actual cluster server, I can change server keys. At least I can debug the scriptlet, which I have to do right now since instances are not starting.

Afterwards, I’ll come back to the remote access.

Getting back to the scriptlet error, I’m getting the following.

On the scriptlet:

def instance_placement(request, candidate_members):
    log_error("SCRIPTLET DEBUG Request: profiles = ", request.profiles, ", name = ", request.name, ", image alias = ", request.source.alias, ", reason = ", request.reason, ", project = ", request.project, "\nSCRIPTLET DEBUG Candidade members: ", [x.server_name for x in candidate_members], "\nSCRIPTLET DEBUG Request config: ", request.config)
    project = get_project( request.project )
    log_error("SCRIPTLET DEBUG Request: ", request, "\nSCRIPTLET DEBUG Candidade members: ", candidate_members, "\nSCRIPTLET DEBUG Project: ", project)

Ran incus launch images:debian/12/cloud teste and on the monitor I get:

ERROR  [2025-07-01T22:39:11-03:00] [amd01] Instance placement scriptlet: SCRIPTLET DEBUG Request: profiles = [], name = teste, image alias = debian/12/cloud, reason = new, project = victor
SCRIPTLET DEBUG Candidade members: ["amd04", "amd01", "amd03", "amd02"]
SCRIPTLET DEBUG Request config: {} 
ERROR  [2025-07-01T22:39:11-03:00] [amd01] Instance placement scriptlet: SCRIPTLET DEBUG Request: profiles = [], name = teste, image alias = debian/12/cloud, reason = new, project = victor
SCRIPTLET DEBUG Candidade members: ["amd04", "amd01", "amd03", "amd02"]
SCRIPTLET DEBUG Request config: {} 
ERROR  [2025-07-01T22:39:11-03:00] [amd01] Instance placement scriptlet: SCRIPTLET DEBUG Request: {"architecture": "x86_64", "config": {}, "devices": {}, "ephemeral": False, "profiles": [], "restore": "", "stateful": False, "description": "", "name": "teste", "source": {"type": "image", "certificate": "", "alias": "debian/12/cloud", "fingerprint": "", "properties": {}, "server": "https://images.linuxcontainers.org", "secret": "", "protocol": "simplestreams", "base-image": "", "mode": "pull", "operation": "", "secrets": {}, "source": "", "live": False, "instance_only": False, "refresh": False, "refresh_exclude_older": False, "project": "", "allow_inconsistent": False}, "instance_type": "", "type": "container", "start": True, "reason": "new", "project": "victor"}
SCRIPTLET DEBUG Candidade members: [{"roles": ["database"], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-5,8-13", "user.experimentos.limits.memory": "24GB"}, "groups": ["default", "amd-5700g"], "server_name": "amd04", "url": "https://10.11.16.14:8443", "database": True, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}, {"roles": [], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-5,8-13", "user.experimentos.limits.memory": "24GB"}, "groups": ["default", "amd-5700g"], "server_name": "amd01", "url": "https://10.11.16.11:8443", "database": False, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}, {"roles": ["database"], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-5,8-13", "user.experimentos.limits.memory": "24GB"}, "groups": ["default", "amd-5700g"], "server_name": "amd03", "url": "https://10.11.16.13:8443", "database": True, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}, {"roles": ["database-leader", "database"], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-5,8-13", "user.experimentos.limits.memory": "24GB"}, "groups": ["default", "amd-5700g"], "server_name": "amd02", "url": "https://10.11.16.12:8443", "database": True, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}]
SCRIPTLET DEBUG Project: {"config": {"features.images": "false", "features.profiles": "true", "features.storage.buckets": "true", "features.storage.volumes": "true", "restricted": "true", "restricted.backups": "allow", "restricted.cluster.groups": "amd-5700g", "restricted.cluster.target": "allow", "restricted.containers.interception": "allow", "restricted.containers.lowlevel": "allow", "restricted.containers.nesting": "allow", "restricted.containers.privilege": "allow", "restricted.devices.nic": "allow", "restricted.snapshots": "allow", "user.node.limits.cpu": "6-7,14-15"}, "description": "Testar serviços", "name": "victor", "used_by": []} 
ERROR  [2025-07-01T22:39:11-03:00] [amd01] Instance placement scriptlet: SCRIPTLET DEBUG Request: {"architecture": "x86_64", "config": {}, "devices": {}, "ephemeral": False, "profiles": [], "restore": "", "stateful": False, "description": "", "name": "teste", "source": {"type": "image", "certificate": "", "alias": "debian/12/cloud", "fingerprint": "", "properties": {}, "server": "https://images.linuxcontainers.org", "secret": "", "protocol": "simplestreams", "base-image": "", "mode": "pull", "operation": "", "secrets": {}, "source": "", "live": False, "instance_only": False, "refresh": False, "refresh_exclude_older": False, "project": "", "allow_inconsistent": False}, "instance_type": "", "type": "container", "start": True, "reason": "new", "project": "victor"}
SCRIPTLET DEBUG Candidade members: [{"roles": ["database"], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-5,8-13", "user.experimentos.limits.memory": "24GB"}, "groups": ["default", "amd-5700g"], "server_name": "amd04", "url": "https://10.11.16.14:8443", "database": True, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}, {"roles": [], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-5,8-13", "user.experimentos.limits.memory": "24GB"}, "groups": ["default", "amd-5700g"], "server_name": "amd01", "url": "https://10.11.16.11:8443", "database": False, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}, {"roles": ["database"], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-5,8-13", "user.experimentos.limits.memory": "24GB"}, "groups": ["default", "amd-5700g"], "server_name": "amd03", "url": "https://10.11.16.13:8443", "database": True, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}, {"roles": ["database-leader", "database"], "failure_domain": "default", "description": "", "config": {"user.experimentos.limits.cpu": "0-5,8-13", "user.experimentos.limits.memory": "24GB"}, "groups": ["default", "amd-5700g"], "server_name": "amd02", "url": "https://10.11.16.12:8443", "database": True, "status": "Online", "message": "Fully operational", "architecture": "x86_64"}]
SCRIPTLET DEBUG Project: {"config": {"features.images": "false", "features.profiles": "true", "features.storage.buckets": "true", "features.storage.volumes": "true", "restricted": "true", "restricted.backups": "allow", "restricted.cluster.groups": "amd-5700g", "restricted.cluster.target": "allow", "restricted.containers.interception": "allow", "restricted.containers.lowlevel": "allow", "restricted.containers.nesting": "allow", "restricted.containers.privilege": "allow", "restricted.devices.nic": "allow", "restricted.snapshots": "allow", "user.node.limits.cpu": "6-7,14-15"}, "description": "Testar serviços", "name": "victor", "used_by": []} 

Not sure why the scriptlet was called twice, but the instance configuration is not going through, for some reason.