Failing to create monitor cgroups after upgrading to incus 7.2

arches-mding · July 24, 2026, 3:24pm

Hi all, I have a 3 server cluster previously running incus v6 and just upgraded to 7.2(1:6.23-debian13-202604030236 → 1:7.2-debian13-202607191816)After upgrading, we got complains from users around super slow ssh into containers, I tried to restart one of the containers and they since fails to come back up.The container in question fails to with cgroup errors, and any copies of that container also fails with similar looking failure to get monitoring cgroup.

… 997 lines of previous errors just like this one …
lxc COPIED_BROKEN_LXC 20260724135113.608 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - Device or resource busy - Failed to create monitor cgroup 10(lxc.monitor.Copied-broken-container-name-998)

lxc Copied-broken-container-name 20260724135113.609 ERROR    cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:483 - Device or resource busy - Failed to create 10(lxc.monitor.Copied-broken-container-name-999)

lxc Copied-broken-container-name 20260724135113.609 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - Device or resource busy - Failed to create monitor cgroup 10(lxc.monitor.Copied-broken-container-name-999)

lxc Copied-broken-container-name 20260724135113.609 ERROR    cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_create:1439 - Numerical result out of range - Failed to create monitor cgroup

lxc Copied-broken-container-name 20260724135113.609 ERROR    start - ../src/lxc/start.c:__lxc_start:2223 - Failed to create monitor cgroup

lxc Copied-broken-container-name 20260724135113.609 ERROR    lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:837 - Received container state “ABORTING” instead of “RUNNING”

lxc Copied-broken-container-name 20260724135113.675 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_payload_destroy:422 - Uninitialized limit cgroup

lxc Copied-broken-container-name 20260724135113.675 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_destroy:623 - Uninitialized monitor cgroup

lxc 20260724135113.675 ERROR    af_unix - ../src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response

lxc 20260724135113.675 ERROR    commands - ../src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command “get_init_pid”

I have tried changing names of the containers to get different cgroup names and the same issue occurs.

############

I have also tried to create new containers using “incus launch” instead of copying the existing container. Newly launched containers are weirdly broken, where i can not exec into the container and it might be in some stuck state:

mding@lxc01:~$ incus exec newly-created-cotainer bash
Error: Failed to retrieve PID of executing child process
mding@lxc01:~$ incus info --show-log newly-created-cotainer
Name: newly-created-cotainer
Description:
Status: RUNNING
Type: container
Architecture: x86_64
Location: lxc01
PID: 3172437
Created: 2026/07/24 10:32 EDT
Last Used: 2026/07/24 11:14 EDT
Started: 2026/07/24 11:14 EDT

Resources:
Processes: 1
Disk usage:
root: 516.00KiB
CPU usage:
CPU usage (in seconds): 0
Memory usage:
Memory (current): 12.33MiB
Network usage:
eth0:
Type: broadcast
State: UP
Host interface: enp33s0f0np0
MAC address: 10:66:6a:be:05:c4
MTU: 1500
Bytes received: 55.76kB
Bytes sent: 936B
Packets received: 769
Packets sent: 12
IP addresses:
inet6: fe80::1266:6aff:febe:5c4/64 (link)
lo:
Type: loopback
State: UP
MTU: 65536
Bytes received: 0B
Bytes sent: 0B
Packets received: 0
Packets sent: 0
IP addresses:
inet:  127.0.0.1/8 (local)
inet6: ::1/128 (local)

Log (lxc.log):

lxc newly-created-cotainer 20260724151434.371 ERROR    cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_attach_create_leaf:2352 - Device or resource busy - Failed to create leaf cgroup “.lxc”
lxc newly-created-cotainer 20260724151434.371 ERROR    conf - ../src/lxc/conf.c:userns_exec_minimal:4538 - Device or resource busy - Running function in new user namespace failed
lxc newly-created-cotainer 20260724151434.371 ERROR    cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_attach_move_into_leaf:2419 - No such file or directory - Failed to open “.lxc/cgroup.procs”
lxc newly-created-cotainer 20260724151434.371 ERROR    conf - ../src/lxc/conf.c:userns_exec_minimal:4576 - No such file or directory - Running parent function failed
lxc newly-created-cotainer 20260724151434.372 ERROR    attach - ../src/lxc/attach.c:do_attach:1242 - No data available - Failed to receive lsm label fd
lxc newly-created-cotainer 20260724151434.372 ERROR    attach - ../src/lxc/attach.c:do_attach:1380 - Failed to attach to container
lxc newly-created-cotainer 20260724152208.170 ERROR    cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_attach_create_leaf:2352 - Device or resource busy - Failed to create leaf cgroup “.lxc”
lxc newly-created-cotainer 20260724152208.170 ERROR    conf - ../src/lxc/conf.c:userns_exec_minimal:4538 - Device or resource busy - Running function in new user namespace failed
lxc newly-created-cotainer 20260724152208.170 ERROR    cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_attach_move_into_leaf:2419 - No such file or directory - Failed to open “.lxc/cgroup.procs”
lxc newly-created-cotainer 20260724152208.170 ERROR    conf - ../src/lxc/conf.c:userns_exec_minimal:4576 - No such file or directory - Running parent function failed
lxc newly-created-cotainer 20260724152208.172 ERROR    attach - ../src/lxc/attach.c:do_attach:1242 - No data available - Failed to receive lsm label fd
lxc newly-created-cotainer 20260724152208.172 ERROR    attach - ../src/lxc/attach.c:do_attach:1380 - Failed to attach to container

I have been fighting this for good part of a day, and trying to locate the root cause with AI help.

Any guidance on what the issue could be would be greatly appriciated.

I am trying to avoid rebooting the server as there are a lot of active users on the machine.
If that becomes non optional then i have to do it

Thanks,

stgraber · July 24, 2026, 3:55pm

Sounds like you’re somehow dealing with a lot of duplicate cgroups in /sys/fs/cgroup/.
You could try and prune them by hand, trying to figure out which aren’t in use, or if possible, a system reboot would clear them all.

We did have a bug in an earlier version of Incus where every restart of an OCI container would leak an extra cgroup. I don’t know if you’re using OCI containers, but if you are, that could have slowly caused a pile up.

In any case, an excessive number of cgroups could definitely explain the performance and startup/restart issue of your containers, so worth a look.

arches-mding · July 24, 2026, 5:09pm

Hi Stéphane,
Thank you for the reply,

I saw the previous issue with OCI containers that had very similar errors. Seems like that has been fixed in 7.2?
I am not running a OCI container, the containers in question was debian13 from images: repo.

I was separately having some issues with UID mapping was causing container startup to crash. But absent a auto restart mechanism that repeatedly crashes containers, could there be so many leaked cgroups for tries for 1000 groups to all fail?
I am not aware of any settings that could make a container restart repeatedly, at least i won’t have set one manually.

at the time i also wasn’t able to find a folder for cgroups named
lxc.monitor.<container_name> under /sys/fs/cgroup/
the AIs suggested it as a thing to check, but i didn’t get too far. Maybe i was looking at the wrong place. I’ll try more.

Strangely this issue has went away on it’s own after an hour or so, and users stopped seeing performance degradation also.

Was there some mechanism where leaked cgroups could time out or otherwise be cleaned up by incus / OS?
I did restart incus service many times during debugging (without restarting running containers)

I’m going to dig into this more, thanks for confirming you think this is a cgroup issue.

stgraber · July 24, 2026, 5:54pm

The error you’re showing definitely would line up with a big pile of cgroups under /sys/fs/cgroup. The exact location can vary somewhat from system to system depending on how incusd is started.

And indeed without autorestart, it would take quite a few manual attempts to hit the issue, though it’s possible that what happened is the cgroup creation itself failed, causing a retry with a different name, eventually running out of all the names on a single container start. That would typically suggest something weird going on with the kernel, but it may be possible.

arches-mding · July 24, 2026, 9:24pm

I’m running into this again after the system seem to have “recovered”
It looks like maybe related to sysfs slowness, if that’s a thing.

Bit of a STR:

I have a few containers that runs github actions runners i am using as example
Shut a couple of them down, they take a few seconds
Start them back up one by one, slowly.
I will get one of the following errors:

A: The startup seem to be improper, and i am unable to access the container. The lack of IP addresses indicate to me the startup was never complete. This container will be stuck in this state for at minimum 30 minutes. I haven’t let this sit for longer yet but i will now.

I attempt to connect to it, get cgroup errors.

mding@lxc01:~$ incus exec gh-build-32 – bash
Error: Failed to retrieve PID of executing child process

mding@lxc01:~$ incus info --show-log gh-build-32
Name: gh-build-32
Description:
Status: RUNNING
Type: container
Architecture: x86_64
Location: lxc01
PID: 1729687
Created: 2025/12/03 15:29 EST
Last Used: 2026/07/24 17:07 EDT
Started: 2026/07/24 17:07 EDT

Resources:
Processes: 16
Disk usage:
root: 134.45GiB
CPU usage:
CPU usage (in seconds): 1
Memory usage:
Memory (current): 62.83MiB
Network usage:
eth0:
Type: broadcast
State: UP
Host interface: enp33s0f0np0
MAC address: 10:66:6a:74:e5:a9
MTU: 1500
Bytes received: 59.93kB
Bytes sent: 936B
Packets received: 974
Packets sent: 12
IP addresses:
inet6: fe80::1266:6aff:fe74:e5a9/64 (link)
lo:
Type: loopback
State: UP
MTU: 65536
Bytes received: 0B
Bytes sent: 0B
Packets received: 0
Packets sent: 0
IP addresses:
inet:  127.0.0.1/8 (local)
inet6: ::1/128 (local)

Log (lxc.log):

lxc gh-build-32 20260724210732.460 ERROR    cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_attach_create_leaf:2352 - Device or resource busy - Failed to create leaf cgroup “.lxc”
lxc gh-build-32 20260724210732.460 ERROR    conf - ../src/lxc/conf.c:userns_exec_minimal:4538 - Device or resource busy - Running function in new user namespace failed
lxc gh-build-32 20260724210732.460 ERROR    cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_attach_move_into_leaf:2419 - No such file or directory - Failed to open “.lxc/cgroup.procs”
lxc gh-build-32 20260724210732.460 ERROR    conf - ../src/lxc/conf.c:userns_exec_minimal:4576 - No such file or directory - Running parent function failed
lxc gh-build-32 20260724210732.461 ERROR    attach - ../src/lxc/attach.c:do_attach:1242 - No data available - Failed to receive lsm label fd
lxc gh-build-32 20260724210732.461 ERROR    attach - ../src/lxc/attach.c:do_attach:1380 - Failed to attach to container
lxc gh-build-32 20260724210744.897 ERROR    cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_attach_create_leaf:2352 - Device or resource busy - Failed to create leaf cgroup “.lxc”
lxc gh-build-32 20260724210744.897 ERROR    conf - ../src/lxc/conf.c:userns_exec_minimal:4538 - Device or resource busy - Running function in new user namespace failed
lxc gh-build-32 20260724210744.897 ERROR    cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_attach_move_into_leaf:2419 - No such file or directory - Failed to open “.lxc/cgroup.procs”
lxc gh-build-32 20260724210744.897 ERROR    conf - ../src/lxc/conf.c:userns_exec_minimal:4576 - No such file or directory - Running parent function failed
lxc gh-build-32 20260724210744.899 ERROR    attach - ../src/lxc/attach.c:do_attach:1242 - No data available - Failed to receive lsm label fd
lxc gh-build-32 20260724210744.899 ERROR    attach - ../src/lxc/attach.c:do_attach:1380 - Failed to attach to container
lxc gh-build-32 20260724211302.137 ERROR    cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_attach_create_leaf:2352 - Device or resource busy - Failed to create leaf cgroup “.lxc”
lxc gh-build-32 20260724211302.137 ERROR    conf - ../src/lxc/conf.c:userns_exec_minimal:4538 - Device or resource busy - Running function in new user namespace failed
lxc gh-build-32 20260724211302.138 ERROR    cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_attach_move_into_leaf:2419 - No such file or directory - Failed to open “.lxc/cgroup.procs”
lxc gh-build-32 20260724211302.138 ERROR    conf - ../src/lxc/conf.c:userns_exec_minimal:4576 - No such file or directory - Running parent function failed
lxc gh-build-32 20260724211302.148 ERROR    attach - ../src/lxc/attach.c:do_attach:1242 - No data available - Failed to receive lsm label fd
lxc gh-build-32 20260724211302.149 ERROR    attach - ../src/lxc/attach.c:do_attach:1380 - Failed to attach to container

B: Container fails to start, same as my earlier log, the cgroup process tries, and fails, 1000 times

C: The container seem to startup fine, but interestingly by trying to create the cgroup, a lot of time, but succeeding before hitting 1000. In the case of container 35 here, it tried 110 times, succeeded, and started up

Location: lxc01
PID: 1742966
Created: 2025/12/03 15:30 EST
Last Used: 2026/07/24 17:09 EDT
Started: 2026/07/24 17:09 EDT

Resources:
Processes: 78
Disk usage:
root: 157.80GiB
CPU usage:
CPU usage (in seconds): 6
Memory usage:
Memory (current): 265.25MiB
Network usage:
docker0:
Type: broadcast
State: UP
MAC address: 02:42:d1:8f:9f:be
MTU: 1500
Bytes received: 0B
Bytes sent: 0B
Packets received: 0
Packets sent: 0
IP addresses:
inet:  172.17.0.1/16 (global)
eth0:
Type: broadcast
State: UP
Host interface: enp33s0f0np0
MAC address: 10:66:6a:1e:42:3c
MTU: 1500
Bytes received: 276.20kB
Bytes sent: 44.35kB
Packets received: 1575
Packets sent: 157
IP addresses:
inet:  10.16.11.49/23 (global)
inet6: fe80::1266:6aff:fe1e:423c/64 (link)
lo:
Type: loopback
State: UP
MTU: 65536
Bytes received: 2.58kB
Bytes sent: 2.58kB
Packets received: 20
Packets sent: 20
IP addresses:
inet:  127.0.0.1/8 (local)
inet6: ::1/128 (local)

Log (lxc.log):
lxc gh-build-35 20260724210916.399 ERROR    cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:483 - Device or resource busy - Failed to create 10(lxc.payload.gh-build-35)
lxc gh-build-35 20260724210916.399 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - Device or resource busy - Failed to create payload cgroup 10(lxc.payload.gh-build-35)
lxc gh-build-35 20260724210916.400 ERROR    cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:483 - Device or resource busy - Failed to create 10(lxc.payload.gh-build-35-1)
lxc gh-build-35 20260724210916.400 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - Device or resource busy - Failed to create payload cgroup 10(lxc.payload.gh-build-35-1)

…

…
lxc gh-build-35 20260724210916.470 ERROR    cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:483 - Device or resource busy - Failed to create 10(lxc.payload.gh-build-35-110)
lxc gh-build-35 20260724210916.470 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:550 - Device or resource busy - Failed to create payload cgroup 10(lxc.payload.gh-build-35-110)
mding@lxc01:~$ ^C

As to my cgroups, i don’t have big piles of it, just one, with the high suffix kind of like this

root@lxc01:/sys/fs/cgroup# ls | grep gh-bu
lxc.monitor.gh-build-30
lxc.monitor.gh-build-31
lxc.monitor.gh-build-32
lxc.monitor.gh-build-33
lxc.monitor.gh-build-34
lxc.monitor.gh-build-35
lxc.monitor.gh-build-36
lxc.monitor.gh-build-37
lxc.monitor.gh-build-38
lxc.monitor.gh-build-cache-server
lxc.monitor.template-gh-build
lxc.payload.gh-build-30
lxc.payload.gh-build-31
lxc.payload.gh-build-32
lxc.payload.gh-build-33
lxc.payload.gh-build-34
lxc.payload.gh-build-35-111
lxc.payload.gh-build-36
lxc.payload.gh-build-37
lxc.payload.gh-build-38
lxc.payload.gh-build-cache-server
lxc.payload.template-gh-build

arches-mding · July 24, 2026, 9:38pm

For this system i have swap on, could that possibly be swap related?
The host has 1.5tb of ram, but with even hungrier users.
The system has 3tb of swap available, usually only a few hundred gb is used.
vm.swapyness is 1

stgraber · July 25, 2026, 2:18am

That EBUSY smells like a kernel bug, all that we’re really doing is a mkdir…

@amikhalitsyn