`stale` cgroup inodes?

Each time incus-compose down and incus-compose up is invoked, a new set of cgroup inodes for each service in the stack is created, with an incremented numeric suffix. The “stale” inodes remain until the host is rebooted or manually deleted. Would the incus daemon or systemd have eventually reaped them otherwise?

...
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-1
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-10
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-11
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-12
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-13
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-14
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-15
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-16
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-17
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-18
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-19
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-2
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-20
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-21
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-22
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-23
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-24
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-25
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-26
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-27
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-28
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-29
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-3
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-30
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-31
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-32
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-33
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-34
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-4
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-5
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-6
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-7
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-8
drwxr-xr-x 2 root root 0 Jul 12 13:16 /sys/fs/cgroup/lxc.monitor.immich_server-9
...

Thanks blurry, you told me before of that maybe this is a incus bug.

Do you want to test that with plain incus and verify that it happens there as well then post it public?

You can post it public without testing plain as well as you wish.

Thank you,
René

EDIT: I’m blind…

What I can tell is that I have the same behavior with incus-compose here.

That’d likely be something for @amikhalitsyn to look at. That’s likely an LXC bug as those cgroups should correctly clear on instance stop, allowing for them to be created with the same name on the next start.

The -X syntax is there to deal with edge cases where we may have been unable to destroy the cgroup or where you somehow have two liblxc instances with the same name on the system.

1 Like

This basically what occurs.

test@eq14-3:~/temp$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 26.04 LTS
Release:        26.04
Codename:       resolute

test@eq14-3:~/temp$ incus version
Client version: 7.2
Server version: 7.2

test@eq14-3:~/temp$ incus-compose version
incus-compose version 1.0.0

test@eq14-3:~/temp$ cat docker-compose.yml
services:
  web:
    container_name: web
    image: docker.io/nginx:alpine
    ports:
      - "8080:80"
    volumes:
      - ./html:/usr/share/nginx/html:ro

test@eq14-3:~/temp$ ls -ld /sys/fs/cgroup/lxc.monitor.temp*
ls: cannot access '/sys/fs/cgroup/lxc.monitor.temp*': No such file or directory

test@eq14-3:~/temp$ incus-compose up
ensure   image              docker.io/nginx:alpine                 [done]
ensure   network            default                                [done]
ensure   instance           web                                    [done]
start    instance           web                                    [done]

test@eq14-3:~/temp$ ls -ld /sys/fs/cgroup/lxc.monitor.temp*
drwxr-xr-x 2 root root 0 Jul 12 15:45 /sys/fs/cgroup/lxc.monitor.temp_web

test@eq14-3:~/temp$ incus-compose down
ensure   instance           web                                    [done]
ensure   network            default                                [done]
stop     instance           web                                    [done]
delete   instance           web                                    [done]
delete   network            default                                [done]

test@eq14-3:~/temp$ ls -ld /sys/fs/cgroup/lxc.monitor.temp*
drwxr-xr-x 2 root root 0 Jul 12 15:45 /sys/fs/cgroup/lxc.monitor.temp_web

test@eq14-3:~/temp$ incus-compose up
ensure   image              docker.io/nginx:alpine                 [done]
ensure   network            default                                [done]
ensure   instance           web                                    [done]
start    instance           web                                    [done]

test@eq14-3:~/temp$ ls -ld /sys/fs/cgroup/lxc.monitor.temp*
drwxr-xr-x 2 root root 0 Jul 12 15:45 /sys/fs/cgroup/lxc.monitor.temp_web
drwxr-xr-x 2 root root 0 Jul 12 15:46 /sys/fs/cgroup/lxc.monitor.temp_web-1

test@eq14-3:~/temp$ incus-compose down
ensure   instance           web                                    [done]
ensure   network            default                                [done]
stop     instance           web                                    [done]
delete   instance           web                                    [done]
delete   network            default                                [done]

test@eq14-3:~/temp$ ls -ld /sys/fs/cgroup/lxc.monitor.temp*
drwxr-xr-x 2 root root 0 Jul 12 15:45 /sys/fs/cgroup/lxc.monitor.temp_web
drwxr-xr-x 2 root root 0 Jul 12 15:46 /sys/fs/cgroup/lxc.monitor.temp_web-1

test@eq14-3:~/temp$ incus-compose up
ensure   image              docker.io/nginx:alpine                 [done]
ensure   network            default                                [done]
ensure   instance           web                                    [done]
start    instance           web                                    [done]

test@eq14-3:~/temp$ ls -ld /sys/fs/cgroup/lxc.monitor.temp*
drwxr-xr-x 2 root root 0 Jul 12 15:45 /sys/fs/cgroup/lxc.monitor.temp_web
drwxr-xr-x 2 root root 0 Jul 12 15:46 /sys/fs/cgroup/lxc.monitor.temp_web-1
drwxr-xr-x 2 root root 0 Jul 12 15:47 /sys/fs/cgroup/lxc.monitor.temp_web-2

test@eq14-3:~/temp$ incus-compose down
ensure   instance           web                                    [done]
ensure   network            default                                [done]
stop     instance           web                                    [done]
delete   instance           web                                    [done]
delete   network            default                                [done]

test@eq14-3:~/temp$ ls -ld /sys/fs/cgroup/lxc.monitor.temp*
drwxr-xr-x 2 root root 0 Jul 12 15:45 /sys/fs/cgroup/lxc.monitor.temp_web
drwxr-xr-x 2 root root 0 Jul 12 15:46 /sys/fs/cgroup/lxc.monitor.temp_web-1
drwxr-xr-x 2 root root 0 Jul 12 15:47 /sys/fs/cgroup/lxc.monitor.temp_web-2

1 Like

You write the best reports ever! Not the first time I’ve seen such one from you.

1 Like

Could reproduce it with incus cli standalone but only with a network attached:

$ lsb_release -a
LSB Version:	n/a
Distributor ID:	cachyos
Description:	CachyOS
Release:	rolling
Codename:	n/a

$ incus version 
Client version: 7.2
Server version: 7.2
# Some custom build pre 7.3 one.

$ incus project create stale-cgroups
Project stale-cgroups created

$ INCUS_PROJECT=stale-cgroups incus profile device add default root disk path=/ pool=default
Device root added to default

$ INCUS_PROJECT=stale-cgroups incus network create sc0
Network sc0 created

$ INCUS_PROJECT=stale-cgroups incus profile device add default eth0 nic network=sc0
Device eth0 added to default

$ ls -ld /sys/fs/cgroup/lxc.monitor.stale-cgroups_web*

$ INCUS_PROJECT=stale-cgroups incus launch docker.io:library/busybox:glibc web
Launching web

$ ls -ld /sys/fs/cgroup/lxc.monitor.stale-cgroups_web*
drwxr-xr-x - root 12 Jul 22:07  /sys/fs/cgroup/lxc.monitor.stale-cgroups_web

$ INCUS_PROJECT=stale-cgroups incus rm -f web

$ INCUS_PROJECT=stale-cgroups incus launch docker.io:library/busybox:glibc web
Launching web

$ INCUS_PROJECT=stale-cgroups incus rm -f web

$ ls -ld /sys/fs/cgroup/lxc.monitor.stale-cgroups_web*
drwxr-xr-x - root 12 Jul 22:07  /sys/fs/cgroup/lxc.monitor.stale-cgroups_web
drwxr-xr-x - root 12 Jul 22:07  /sys/fs/cgroup/lxc.monitor.stale-cgroups_web-1