I’m having a particular issue which happens on some particular value of project limits on the cluster I’m setting up.
I’m currently on a project in an incus cluster in which I’ve set a limit on the number of instances to be the number of machines in the cluster which can be seen as follows.
pargo@bastion:~$ incus project list
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-----------------------+---------+
| NAME | IMAGES | PROFILES | STORAGE VOLUMES | STORAGE BUCKETS | NETWORKS | NETWORK ZONES | DESCRIPTION | USED BY |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-----------------------+---------+
| dedicado (current) | YES | YES | YES | YES | NO | NO | | 7 |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-----------------------+---------+
| default | YES | YES | YES | YES | YES | YES | Default Incus project | 1 |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-----------------------+---------+
pargo@bastion:~$ incus cluster list
+------------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE | MESSAGE |
+------------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| dedicado01 | https://10.11.16.31:8443 | database-leader | x86_64 | default | | ONLINE | Fully operational |
| | | database | | | | | |
+------------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| dedicado02 | https://10.11.16.32:8443 | database | x86_64 | default | | ONLINE | Fully operational |
+------------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| dedicado03 | https://10.11.16.33:8443 | database | x86_64 | default | | ONLINE | Fully operational |
+------------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
pargo@bastion:~$ incus project show dedicado
config:
features.images: "true"
features.profiles: "true"
features.storage.buckets: "true"
features.storage.volumes: "true"
limits.instances: "3"
description: ""
name: dedicado
used_by:
- /1.0/instances/d1?project=dedicado
- /1.0/instances/d2?project=dedicado
- /1.0/instances/d3?project=dedicado
- /1.0/instances/d4?project=dedicado
- /1.0/profiles/default?project=dedicado
- /1.0/images/35713da17c2971321aac85b7a4f8ca2bdb742ee0ee8235a63d8954636c966682?project=dedicado
- /1.0/images/6325d25ee7ba236f1cc63d3eb068d87bba98c74400cf112f353f9f35cb06f818?project=dedicado
The issue I’m having is that I can create 4 instances and the fifth instance in blocked.
pargo@bastion:~$ incus list
+------+---------+--------------------+------+-----------+-----------+------------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+---------+--------------------+------+-----------+-----------+------------+
| d1 | RUNNING | 10.11.21.27 (eth0) | | CONTAINER | 0 | dedicado01 |
+------+---------+--------------------+------+-----------+-----------+------------+
| d2 | RUNNING | 10.11.21.28 (eth0) | | CONTAINER | 0 | dedicado02 |
+------+---------+--------------------+------+-----------+-----------+------------+
| d3 | RUNNING | 10.11.21.29 (eth0) | | CONTAINER | 0 | dedicado03 |
+------+---------+--------------------+------+-----------+-----------+------------+
| d4 | RUNNING | 10.11.21.30 (eth0) | | CONTAINER | 0 | dedicado01 |
+------+---------+--------------------+------+-----------+-----------+------------+
pargo@bastion:~$ incus launch images:debian/12 d5
Launching d5
Error: Failed instance creation: Reached maximum number of instances in project "dedicado"
What’s particularly awkward is that the same limit works on some values, but not on others.
Doesn’t work on 0 or 3, but works on 1 and 2.
For limits.instances=0.
pargo@bastion:~$ incus delete -f d1 d2 d3 d4
pargo@bastion:~$ incus list
+------+-------+------+------+------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+-------+------+------+------+-----------+----------+
pargo@bastion:~$ incus project set dedicado limits.instances=0
pargo@bastion:~$ incus launch images:debian/12 d1
Launching d1
pargo@bastion:~$ incus launch images:debian/12 d2
Launching d2
Error: Failed instance creation: Reached maximum number of instances in project "dedicado"
For limits.instances=1.
pargo@bastion:~$ incus delete -f d1
pargo@bastion:~$ incus list
+------+-------+------+------+------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+-------+------+------+------+-----------+----------+
pargo@bastion:~$ incus project set dedicado limits.instances=1
pargo@bastion:~$ incus launch images:debian/12 d1
Launching d1
pargo@bastion:~$ incus launch images:debian/12 d2
Launching d2
Error: Failed instance creation: Reached maximum number of instances in project "dedicado"
For limits.instances=2.
pargo@bastion:~$ incus delete -f d1
pargo@bastion:~$ incus list
+------+-------+------+------+------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+-------+------+------+------+-----------+----------+
pargo@bastion:~$ incus project set dedicado limits.instances=2
pargo@bastion:~$ incus launch images:debian/12 d1
Launching d1
pargo@bastion:~$ incus launch images:debian/12 d2
Launching d2
pargo@bastion:~$ incus launch images:debian/12 d3
Launching d3
Error: Failed instance creation: Reached maximum number of instances in project "dedicado"
For limits.instances=3.
pargo@bastion:~$ incus delete -f d1 d2
pargo@bastion:~$ incus list
+------+-------+------+------+------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+-------+------+------+------+-----------+----------+
pargo@bastion:~$ incus project set dedicado limits.instances=3
pargo@bastion:~$ incus launch images:debian/12 d1
Launching d1
pargo@bastion:~$ incus launch images:debian/12 d2
Launching d2
pargo@bastion:~$ incus launch images:debian/12 d3
Launching d3
pargo@bastion:~$ incus launch images:debian/12 d4
Launching d4
pargo@bastion:~$ incus launch images:debian/12 d5
Launching d5
Error: Failed instance creation: Reached maximum number of instances in project "dedicado"
I did mention this is a cluster as I could not reproduce this issue in the “Try it online” version of incus and the issue seems to happen when the limit is set as a multiple of the number of machines in the cluster.
My guess is related to the cluster machine scheduler since it schedules the instances cycling through the machines. Whenever it is about to start a new cycle, it starts the instance without checking project limits.
To see that this makes sense, I’ll create an instance on the default project and set limits.instances=3
just so that the scheduler cycling does not occur on the fourth instance. In this case, the project limit is verified and the instance is not created.
pargo@bastion:~$ incus list
+------+-------+------+------+------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+-------+------+------+------+-----------+----------+
pargo@bastion:~$ incus project set dedicado limits.instances=3
pargo@bastion:~$ incus init images:debian/12 d0 --project default
Creating d0
pargo@bastion:~$ incus init images:debian/12 d1
Creating d1
pargo@bastion:~$ incus init images:debian/12 d2
Creating d2
pargo@bastion:~$ incus init images:debian/12 d3
Creating d3
pargo@bastion:~$ incus init images:debian/12 d4
Creating d4
Error: Failed instance creation: Reached maximum number of instances in project "dedicado"
pargo@bastion:~$ incus list
+------+---------+------+------+-----------+-----------+------------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+---------+------+------+-----------+-----------+------------+
| d1 | STOPPED | | | CONTAINER | 0 | dedicado02 |
+------+---------+------+------+-----------+-----------+------------+
| d2 | STOPPED | | | CONTAINER | 0 | dedicado03 |
+------+---------+------+------+-----------+-----------+------------+
| d3 | STOPPED | | | CONTAINER | 0 | dedicado01 |
+------+---------+------+------+-----------+-----------+------------+