Incus project limits.instances awkward behaviour in cluster

victoitor · February 2, 2024, 2:20pm

I’m having a particular issue which happens on some particular value of project limits on the cluster I’m setting up.

I’m currently on a project in an incus cluster in which I’ve set a limit on the number of instances to be the number of machines in the cluster which can be seen as follows.

pargo@bastion:~$ incus project list
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-----------------------+---------+
|        NAME        | IMAGES | PROFILES | STORAGE VOLUMES | STORAGE BUCKETS | NETWORKS | NETWORK ZONES |      DESCRIPTION      | USED BY |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-----------------------+---------+
| dedicado (current) | YES    | YES      | YES             | YES             | NO       | NO            |                       | 7       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-----------------------+---------+
| default            | YES    | YES      | YES             | YES             | YES      | YES           | Default Incus project | 1       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-----------------------+---------+
pargo@bastion:~$ incus cluster list
+------------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
|    NAME    |           URL            |      ROLES      | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+------------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| dedicado01 | https://10.11.16.31:8443 | database-leader | x86_64       | default        |             | ONLINE | Fully operational |
|            |                          | database        |              |                |             |        |                   |
+------------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| dedicado02 | https://10.11.16.32:8443 | database        | x86_64       | default        |             | ONLINE | Fully operational |
+------------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| dedicado03 | https://10.11.16.33:8443 | database        | x86_64       | default        |             | ONLINE | Fully operational |
+------------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
pargo@bastion:~$ incus project show dedicado
config:
  features.images: "true"
  features.profiles: "true"
  features.storage.buckets: "true"
  features.storage.volumes: "true"
  limits.instances: "3"
description: ""
name: dedicado
used_by:
- /1.0/instances/d1?project=dedicado
- /1.0/instances/d2?project=dedicado
- /1.0/instances/d3?project=dedicado
- /1.0/instances/d4?project=dedicado
- /1.0/profiles/default?project=dedicado
- /1.0/images/35713da17c2971321aac85b7a4f8ca2bdb742ee0ee8235a63d8954636c966682?project=dedicado
- /1.0/images/6325d25ee7ba236f1cc63d3eb068d87bba98c74400cf112f353f9f35cb06f818?project=dedicado

The issue I’m having is that I can create 4 instances and the fifth instance in blocked.

pargo@bastion:~$ incus list
+------+---------+--------------------+------+-----------+-----------+------------+
| NAME |  STATE  |        IPV4        | IPV6 |   TYPE    | SNAPSHOTS |  LOCATION  |
+------+---------+--------------------+------+-----------+-----------+------------+
| d1   | RUNNING | 10.11.21.27 (eth0) |      | CONTAINER | 0         | dedicado01 |
+------+---------+--------------------+------+-----------+-----------+------------+
| d2   | RUNNING | 10.11.21.28 (eth0) |      | CONTAINER | 0         | dedicado02 |
+------+---------+--------------------+------+-----------+-----------+------------+
| d3   | RUNNING | 10.11.21.29 (eth0) |      | CONTAINER | 0         | dedicado03 |
+------+---------+--------------------+------+-----------+-----------+------------+
| d4   | RUNNING | 10.11.21.30 (eth0) |      | CONTAINER | 0         | dedicado01 |
+------+---------+--------------------+------+-----------+-----------+------------+
pargo@bastion:~$ incus launch images:debian/12 d5
Launching d5
Error: Failed instance creation: Reached maximum number of instances in project "dedicado"

What’s particularly awkward is that the same limit works on some values, but not on others.

Doesn’t work on 0 or 3, but works on 1 and 2.

For limits.instances=0.

pargo@bastion:~$ incus delete -f d1 d2 d3 d4
pargo@bastion:~$ incus list
+------+-------+------+------+------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+-------+------+------+------+-----------+----------+
pargo@bastion:~$ incus project set dedicado limits.instances=0
pargo@bastion:~$ incus launch images:debian/12 d1
Launching d1
pargo@bastion:~$ incus launch images:debian/12 d2
Launching d2
Error: Failed instance creation: Reached maximum number of instances in project "dedicado"

For limits.instances=1.

pargo@bastion:~$ incus delete -f d1
pargo@bastion:~$ incus list
+------+-------+------+------+------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+-------+------+------+------+-----------+----------+
pargo@bastion:~$ incus project set dedicado limits.instances=1
pargo@bastion:~$ incus launch images:debian/12 d1
Launching d1
pargo@bastion:~$ incus launch images:debian/12 d2
Launching d2
Error: Failed instance creation: Reached maximum number of instances in project "dedicado"

For limits.instances=2.

pargo@bastion:~$ incus delete -f d1
pargo@bastion:~$ incus list
+------+-------+------+------+------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+-------+------+------+------+-----------+----------+
pargo@bastion:~$ incus project set dedicado limits.instances=2
pargo@bastion:~$ incus launch images:debian/12 d1
Launching d1
pargo@bastion:~$ incus launch images:debian/12 d2
Launching d2
pargo@bastion:~$ incus launch images:debian/12 d3
Launching d3
Error: Failed instance creation: Reached maximum number of instances in project "dedicado"

For limits.instances=3.

pargo@bastion:~$ incus delete -f d1 d2
pargo@bastion:~$ incus list
+------+-------+------+------+------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+-------+------+------+------+-----------+----------+
pargo@bastion:~$ incus project set dedicado limits.instances=3
pargo@bastion:~$ incus launch images:debian/12 d1
Launching d1
pargo@bastion:~$ incus launch images:debian/12 d2
Launching d2
pargo@bastion:~$ incus launch images:debian/12 d3
Launching d3
pargo@bastion:~$ incus launch images:debian/12 d4
Launching d4
pargo@bastion:~$ incus launch images:debian/12 d5
Launching d5
Error: Failed instance creation: Reached maximum number of instances in project "dedicado"

I did mention this is a cluster as I could not reproduce this issue in the “Try it online” version of incus and the issue seems to happen when the limit is set as a multiple of the number of machines in the cluster.

My guess is related to the cluster machine scheduler since it schedules the instances cycling through the machines. Whenever it is about to start a new cycle, it starts the instance without checking project limits.

To see that this makes sense, I’ll create an instance on the default project and set limits.instances=3 just so that the scheduler cycling does not occur on the fourth instance. In this case, the project limit is verified and the instance is not created.

pargo@bastion:~$ incus list
+------+-------+------+------+------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+-------+------+------+------+-----------+----------+
pargo@bastion:~$ incus project set dedicado limits.instances=3
pargo@bastion:~$ incus init images:debian/12 d0 --project default
Creating d0
pargo@bastion:~$ incus init images:debian/12 d1
Creating d1
pargo@bastion:~$ incus init images:debian/12 d2
Creating d2
pargo@bastion:~$ incus init images:debian/12 d3
Creating d3
pargo@bastion:~$ incus init images:debian/12 d4
Creating d4
Error: Failed instance creation: Reached maximum number of instances in project "dedicado"
pargo@bastion:~$ incus list
+------+---------+------+------+-----------+-----------+------------+
| NAME |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |  LOCATION  |
+------+---------+------+------+-----------+-----------+------------+
| d1   | STOPPED |      |      | CONTAINER | 0         | dedicado02 |
+------+---------+------+------+-----------+-----------+------------+
| d2   | STOPPED |      |      | CONTAINER | 0         | dedicado03 |
+------+---------+------+------+-----------+-----------+------------+
| d3   | STOPPED |      |      | CONTAINER | 0         | dedicado01 |
+------+---------+------+------+-----------+-----------+------------+

victoitor · February 2, 2024, 2:22pm

Just tested and it works on 4 and 5 and fails again on 6 by 1, just like for 0 and 3.

stgraber · February 2, 2024, 2:25pm

Definitely sounds like a bug, can you report this at Issues · lxc/incus · GitHub?

victoitor · February 2, 2024, 2:26pm

I will and I’ll add a bit more testing that I’m trying to figure out what it is.

victoitor · February 2, 2024, 3:56pm

Posted.

Can you point me in the general direction of where I can find the following in the incus code?

Where project limits are verified when starting an instance.
Where the scheduler code is for deciding in which machine an instance will run.
Who calls this scheduler code.

I still have a lot of trouble finding incus source code and I’ve been trying to figure these out and still have not found them.

stgraber · February 2, 2024, 4:14pm

github.com

lxc/incus/blob/main/internal/server/project/permissions.go#L26


      
          	"github.com/lxc/incus/internal/server/instance/instancetype"
          	"github.com/lxc/incus/shared/api"
          	"github.com/lxc/incus/shared/idmap"
          	"github.com/lxc/incus/shared/units"
          	"github.com/lxc/incus/shared/util"
          	"github.com/lxc/incus/shared/validate"
          )
          
          // AllowInstanceCreation returns an error if any project-specific limit or
          // restriction is violated when creating a new instance.
          func AllowInstanceCreation(tx *db.ClusterTx, projectName string, req api.InstancesPost) error {
          	info, err := fetchProject(tx, projectName, true)
          	if err != nil {
          		return err
          	}
          
          	if info == nil {
          		return nil
          	}
          
          	var instanceType instancetype.Type

github.com

lxc/incus/blob/main/cmd/incusd/instances_post.go#L899


      
          	}
          
          	var targetProject *api.Project
          	var profiles []api.Profile
          	var sourceInst *dbCluster.Instance
          	var sourceImage *api.Image
          	var sourceImageRef string
          	var candidateMembers []db.NodeInfo
          	var targetMemberInfo *db.NodeInfo
          
          	err = s.DB.Cluster.Transaction(r.Context(), func(ctx context.Context, tx *db.ClusterTx) error {
          		target := request.QueryParam(r, "target")
          		if !s.ServerClustered && target != "" {
          			return api.StatusErrorf(http.StatusBadRequest, "Target only allowed when clustered")
          		}
          
          		dbProject, err := dbCluster.GetProject(ctx, tx.Tx(), targetProjectName)
          		if err != nil {
          			return fmt.Errorf("Failed loading project: %w", err)
          		}

The function above does, it first validates the target per project config, then looks for a number of candidates and even calls the placement scriplet if one is in place.

After that all runs, we know what server needs to process the request and the request is forwarded over to it.