Could someone explain to me the instance autostart options, particularly the order in which containers start when some but not all of these options are set?
To did a quick experiment with the following 6 containers. (I used OCI containers as it easier to monitor when they start using oci.entrypoint)
~ % incus launch docker:alpine c1 --target incus-os2 -c oci.entrypoint="sh -c 'date && sleep infinity'" -c boot.autostart=true -c boot.autostart.priority=100 -c boot.autostart.delay=10
Launching c1
~ % incus launch docker:alpine c2 --target incus-os2 -c oci.entrypoint="sh -c 'date && sleep infinity'" -c boot.autostart=true -c boot.autostart.priority=50 -c boot.autostart.delay=5
Launching c2
~ % incus launch docker:alpine c3 --target incus-os2 -c oci.entrypoint="sh -c 'date && sleep infinity'" -c boot.autostart=true -c boot.autostart.delay=5
Launching c3
~ % incus launch docker:alpine c4 --target incus-os2 -c oci.entrypoint="sh -c 'date && sleep infinity'" -c boot.autostart=true
Launching c4
~ % incus launch docker:alpine c5 --target incus-os2 -c oci.entrypoint="sh -c 'date && sleep infinity'" -c boot.autostart.delay=5
Launching c5
~ % incus launch docker:alpine c6 --target incus-os2 -c oci.entrypoint="sh -c 'date && sleep infinity'"
Launching c6
In my undestanding, container c1 should start first because it has the highest priority, then after 10 seconds dealy c2 should start, after another 5 seconds container c3-c6 (in arbitrary order) because they have the same default priority 0, where there should be a dealy of 5 seconds after starting c3 and c5.
However, here is what I observe:
~ % incus admin os system reboot --target incus-os2
WARNING: The IncusOS API and configuration is subject to change
Are you sure you want to reboot the system? (yes/no) [default=no]: yes
~ % incus console c1 --show-log
Tue Feb 3 13:26:57 UTC 2026
~ % incus console c2 --show-log
Tue Feb 3 13:27:08 UTC 2026
~ % incus console c3 --show-log
Tue Feb 3 13:27:14 UTC 2026
~ % incus console c4 --show-log
Tue Feb 3 13:26:57 UTC 2026
~ % incus console c5 --show-log
Tue Feb 3 13:27:20 UTC 2026
~ % incus console c6 --show-log
Tue Feb 3 13:26:56 UTC 2026
Looks like containers start in the order: c1+c4+c6, c2, c3, c5. (I consider the containers starting with ± 1 second delay as starting at the same time.)
Looks like this is not a coincidence. Upon the second reboot I obtain the same order:
~ % incus admin os system reboot --target incus-os2
WARNING: The IncusOS API and configuration is subject to change
Are you sure you want to reboot the system? (yes/no) [default=no]: yes
~ % incus console c1 --show-log
Tue Feb 3 13:33:08 UTC 2026
~ % incus console c2 --show-log
Tue Feb 3 13:33:20 UTC 2026
~ % incus console c3 --show-log
Tue Feb 3 13:33:26 UTC 2026
~ % incus console c4 --show-log
Tue Feb 3 13:33:07 UTC 2026
~ % incus console c5 --show-log
Tue Feb 3 13:33:32 UTC 2026
~ % incus console c6 --show-log
Tue Feb 3 13:33:08 UTC 2026
Particularly, it is puzzling why c4 starts before c2 but c3 after. Both c3 and c4 have the default priority 0 so they should start after all containers with the higher priorities. Also the relative starting order of containes with and without boot.autostart option set is not clear from the documentation or from this experiment. For example does setting of the options boot.autostart.priority or boot.autostart.delay have any effect if boot.autostart is not set?
Background: I am trying to setup services for IncusOS applications incus-ovn, incus-ceph, and incus-linstor winthin containers, and try to ensure that these services start before any other container that that depend on these services. I was hoping that containers that do not use any auto-start options are started after all containers with auto-start options, but this does not appear to be the case as the above example shows (unles there is a bug). This means that I should probably modify the default profile somehow to achieve the desired behaviour.
I am using a test IncusOS (version 202602030721) cluster installed inside VMs.
In my understanding, the behaviour should not be a result of the recent changes of “Parallel instance startup” in Incus 6.21, since the default project in which all instances run have instances with auto-start enabled.