Limitations not applied inside of containers (aarch64 & sysvinit system)

jean-marc-LACROIX · March 13, 2025, 9:30am

Dear maintainers,

I a trying to use limitation cgroup features on CPU and memory on a
pure cgroupv2 system.

The target system is a Rock64 SBC (ARM64 architecture) with 4 GB of
RAM, a 16 GB SD card, and a 32 GB MMC card.

The current version is Debian 12.9 (Bookworm) with the latest Debian
Trixie Linux kernel.

ansible@hn-rock64-130:~$ cat /etc/debian_version
12.9

ansible@hn-rock64-130:~$ dpkg -l |grep linux-image
ii linux-image-6.12.12-arm64 6.12.12-1 arm64 Linux 6.12 for 64-bit ARMv8 machines (signed)
ii linux-image-arm64 6.12.12-1 arm64 Linux for 64-bit ARMv8 machines (meta-package)

ansible@hn-rock64-130:~$ dpkg -l |grep lxc
ii liblxc-common 1:5.0.2-1+deb12u3 arm64 Linux Containers userspace tools (library)
ii liblxc1:arm64 1:5.0.2-1+deb12u3 arm64 Linux Containers userspace tools (library)
ii lxc 1:5.0.2-1+deb12u3 arm64 Linux Containers userspace tools

Init system use sysvinit, no systemd. Therefore, in order to be sure
to boot one pure cgroupv2 system, kernel is booting (via flashkernel
Debian tools) wit following args

ansible@hn-rock64-130:~$ cat /proc/cmdline
UUID=df5f524b-9e81-463b-946b-fcf9034f7cb3 log_buf_len=1M net.ifnames=0 fsck.mode=force fsck.repair=yes ipv6.disable=1 loglevel=7 apparmor=0 selinux=0 cgroup_no_v1=all

Furthermore, /etc/fstab is update from brevious old cgroup V1 with
following lines

ansible@hn-rock64-130:~$ grep cgroup /etc/fstab
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime 0 0

Therefore, cgroupV2 seems correctly initialized, because …

ansible@hn-rock64-130:~$ stat -fc %T /sys/fs/cgroup/
cgroup2fs

…and

ansible@hn-rock64-130:~$ mount |grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)

ansible@hn-rock64-130:~$ lxc-checkconfig
LXC version 5.0.2
Kernel configuration not found at /proc/config.gz; searching…
Kernel configuration found at /boot/config-6.12.12-arm64

— Namespaces —
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled

— Control groups —
Cgroups: enabled
Cgroup namespace: enabled
Cgroup v1 mount points:
Cgroup v2 mount points:

/sys/fs/cgroup
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

— Misc —
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, loaded

— Checkpoint/Restore —
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

Configuration for one test containter (name : vn-ntp-130) is defined
here …

According LXC documentation, is seems that when using pure cgroupv2
system, it is the mandatory to change prefix cgroup to cgroup2 in
configuration (see above).

Therefore, i make the change on the 2 following lines, but the
container can no more start, log file is bellow

---- change done —
lxc.cgroup2.memory.limit_in_bytes = 410M
lxc.cgroup2.memory.memsw.limit_in_bytes = 410M

— full confiuration file

ansible@hn-rock64-130:~$ sudo cat /etc/lxc/auto/vn-ntp-130 |grep -v “#” |grep -v ^$

lxc.arch = aarch64
lxc.uts.name = vn-ntp-130
lxc.start.auto = 1
lxc.start.order = 10
lxc.start.delay = 0
lxc.group = grp_lxc_start_on_boot
lxc.init.cmd = /sbin/init
lxc.init.uid = 0
lxc.init.gid = 0
lxc.ephemeral = 0

lxc.console.buffer.size = 102400
lxc.console.size = 102400
lxc.log.level = DEBUG
lxc.log.file = /var/log/lxc/vn-ntp-130.log
lxc.tty.max = 4
lxc.pty.max = 10
lxc.signal.halt = SIGPWR
lxc.signal.reboot = SIGINT
lxc.signal.stop = SIGKILL
lxc.cgroup2.memory.limit_in_bytes = 410M
lxc.cgroup2.memory.memsw.limit_in_bytes = 410M

lxc.cgroup.devices.deny = a
lxc.autodev = 1
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 136:0 rwm
lxc.cgroup.devices.allow = c 136:1 rwm
lxc.cgroup.devices.allow = c 136:2 rwm
lxc.cgroup.devices.allow = c 136:3 rwm
lxc.cgroup.devices.allow = c 136:4 rwm
lxc.cgroup.devices.allow = c 136:5 rwm
lxc.cgroup.devices.allow = c 136:6 rwm
lxc.cgroup.devices.allow = c 136:7 rwm
lxc.cgroup.devices.allow = c 136:8 rwm
lxc.cgroup.devices.allow = c 136:9 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 4:64 rwm
lxc.cgroup.devices.allow = c 4:65 rwm
lxc.cgroup.devices.allow = c 4:0 rwm
lxc.cgroup.devices.allow = c 4:1 rwm
lxc.cgroup.devices.allow = c 4:2 rwm
lxc.cgroup.devices.allow = c 4:3 rwm
lxc.cgroup.devices.allow = c 4:4 rwm
lxc.cgroup.devices.allow = c 4:5 rwm
lxc.cgroup.devices.allow = c 4:6 rwm
lxc.rootfs.mount = /var/lib/lxc/vn-ntp-130/rootfs
lxc.rootfs.path = /dev/mapper/vg_vn_ntp_130-lv_rootfs
lxc.rootfs.options = defaults,noatime,nodiratime
lxc.mount.entry = proc /var/lib/lxc/vn-ntp-130/rootfs/proc proc nodev,noexec,nosuid 0 0
lxc.mount.entry = devpts /var/lib/lxc/vn-ntp-130/rootfs/dev/pts devpts defaults 0 0
lxc.mount.entry = sysfs /var/lib/lxc/vn-ntp-130/rootfs/sys sysfs defaults 0 0
lxc.mount.entry = /dev/mapper/vg_vn_ntp_130-lv_usr /var/lib/lxc/vn-ntp-130/rootfs/usr ext4 defaults,noatime,nodiratime
lxc.mount.entry = /dev/mapper/vg_vn_ntp_130-lv_var /var/lib/lxc/vn-ntp-130/rootfs/var ext4 defaults,noatime,nodiratime
lxc.mount.entry = /dev/mapper/vg_vn_ntp_130-lv_tmp /var/lib/lxc/vn-ntp-130/rootfs/tmp ext4 defaults,noatime,nodiratime
lxc.mount.entry = /dev/mapper/vg_vn_ntp_130-lv_home /var/lib/lxc/vn-ntp-130/rootfs/home ext4 defaults,noatime,nodiratime
lxc.mount.entry = /dev/mapper/vg_vn_ntp_130-lv_var_log /var/lib/lxc/vn-ntp-130/rootfs/var/log ext4 defaults,noatime,nodiratime
lxc.mount.entry = /dev/mapper/vg_vn_ntp_130-lv_var_lib /var/lib/lxc/vn-ntp-130/rootfs/var/lib ext4 defaults,noatime,nodiratime
lxc.mount.entry = /dev/mapper/vg_vn_ntp_130-lv_var_cache /var/lib/lxc/vn-ntp-130/rootfs/var/cache ext4 defaults,noatime,nodiratime
lxc.mount.entry = /dev/mapper/vg_vn_ntp_130-lv_var_lib_apt /var/lib/lxc/vn-ntp-130/rootfs/var/lib/apt ext4 defaults,noatime,nodiratime
lxc.cap.drop = sys_ptrace
lxc.cap.drop = mknod
lxc.cap.drop = sys_module
lxc.cap.drop = sys_rawio
lxc.cap.drop = syslog
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = br-admin
lxc.net.0.name = if-admin
lxc.net.0.hwaddr = 02:00:30:70:04:9
lxc.net.0.veth.pair = if-ntp-adm
lxc.net.1.type = veth
lxc.net.1.flags = up
lxc.net.1.link = br-user
lxc.net.1.name = if-user
lxc.net.1.hwaddr = 02:00:30:70:04:8
lxc.net.1.veth.pair = if-ntp-usr
lxc.net.2.type = veth
lxc.net.2.flags = up
lxc.net.2.link = br-ntp
lxc.net.2.name = if-ntp
lxc.net.2.hwaddr = 02:00:30:70:04:7
lxc.net.2.veth.pair = if-ntp-ntp
lxc.net.3.type = veth
lxc.net.3.flags = up
lxc.net.3.link = br-serv
lxc.net.3.name = if-serv
lxc.net.3.hwaddr = 02:00:30:70:04:6
lxc.net.3.veth.pair = if-ntp-srv
lxc.net.4.type = veth
lxc.net.4.flags = up
lxc.net.4.link = br-factory
lxc.net.4.name = if-factory
lxc.net.4.hwaddr = 02:00:30:70:04:15
lxc.net.4.veth.pair = if-ntp-fact
lxc.apparmor.allow_incomplete = 1
lxc.apparmor.profile = unconfined

ansible@hn-rock64-130:~$ tail -f /var/log/lxc/vn-ntp-130.log

lxc-start vn-ntp-130 20250313085530.640 INFO lxc-start vn-ntp-130 20250313085530.641 INFO lxc-start vn-ntp-130 20250313085530.642 INFO lxc-start vn-ntp-130 20250313085530.671 DEBUG lxc-start vn-ntp-130 20250313085530.672 INFO lxc-start vn-ntp-130 20250313085530.675 INFO lxc-start vn-ntp-130 20250313085530.675 INFO lxc-start vn-ntp-130 20250313085530.675 INFO lxc-start vn-ntp-130 20250313085530.675 INFO lxc-start vn-ntp-130 20250313085530.675 INFO lxc-start vn-ntp-130 20250313085530.675 INFO lxc-start vn-ntp-130 20250313085530.675 DEBUG lxc-start vn-ntp-130 20250313085530.675 DEBUG lxc-start vn-ntp-130 20250313085530.676 DEBUG lxc-start vn-ntp-130 20250313085530.676 DEBUG lxc-start vn-ntp-130 20250313085530.676 DEBUG lxc-start vn-ntp-130 20250313085530.676 DEBUG lxc-start vn-ntp-130 20250313085530.677 WARN lxc-start vn-ntp-130 20250313085530.677 ERROR lxc-start vn-ntp-130 20250313085530.677 ERROR lxc-start vn-ntp-130 20250313085530.678 DEBUG lxc-start vn-ntp-130 20250313085530.679 ERROR lxc-start vn-ntp-130 20250313085530.680 WARN lsm - …/src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver nop
start - …/src/lxc/start.c:lxc_init:881 - Container “vn-ntp-130” is initialized
cgfsng - …/src/lxc/cgroups/cgfsng.c:cgfsng_monitor_create:1391 - The monitor process uses “lxc.monitor.vn-ntp-130” as cgroup
storage - …/src/lxc/storage/storage.c:storage_query:231 - Detected rootfs type “lvm”
cgfsng - …/src/lxc/cgroups/cgfsng.c:cgfsng_payload_create:1499 - The container process uses “lxc.payload.vn-ntp-130” as inner and “lxc.payload.vn-ntp-130” as limit cgroup
start - …/src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWNS
start - …/src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWPID
start - …/src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWUTS
start - …/src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWIPC
start - …/src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWNET
start - …/src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWCGROUP
start - …/src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved mnt namespace via fd 16 and stashed path as mnt:/proc/19844/fd/16
start - …/src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved pid namespace via fd 17 and stashed path as pid:/proc/19844/fd/17
start - …/src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved uts namespace via fd 18 and stashed path as uts:/proc/19844/fd/18
start - …/src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved ipc namespace via fd 19 and stashed path as ipc:/proc/19844/fd/19
start - …/src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved net namespace via fd 20 and stashed path as net:/proc/19844/fd/20
start - …/src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved cgroup namespace via fd 21 and stashed path as cgroup:/proc/19844/fd/21
cgfsng - …/src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits_legacy:3155 - Invalid argument - Ignoring legacy cgroup limits on pure cgroup2 system
cgfsng - …/src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits:3246 - No such file or directory - Failed to set “memory.limit_in_bytes” to “410M”
start - …/src/lxc/start.c:lxc_spawn:1810 - Failed to setup cgroup limits for container “vn-ntp-130”
network - …/src/lxc/network.c:lxc_delete_network:4173 - Deleted network devices
start - …/src/lxc/start.c:__lxc_start:2107 - Failed to spawn container “vn-ntp-130”
start - …/src/lxc/start.c:lxc_abort:1036 - No such process - Failed to send SIGKILL via pidfd 15 for process 19846

Thank in advance for your help

PS: i have also try to limit cpu, but without any succes

stgraber · March 13, 2025, 5:23pm

How did you check that the container doesn’t have the limit applied?

jean-marc-LACROIX · March 15, 2025, 10:24am

When using following lines into configuration file …
lxc.cgroup2.memory.limit_in_bytes = 410M
lxc.cgroup2.memory.memsw.limit_in_bytes = 410M

Then not possible to boot the container

stgraber · March 15, 2025, 6:16pm

What’s the output of ls -lh /sys/fs/cgroup/?

jean-marc-LACROIX · March 15, 2025, 9:17pm

ansible@hn-rock64-130:~$ cat /etc/fstab
UUID=df5f524b-9e81-463b-946b-fcf9034f7cb3 / ext4 noatime,nodiratime,errors=remount-ro 0 1
debugfs /sys/kernel/debug/ debugfs 0 0
/dev/mmcblk0p5 none swap sw 0 0
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime 0 0
ansible@hn-rock64-130:~$
ansible@hn-rock64-130:~$ mount |grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)

ansible@hn-rock64-130:~$ cat /proc/cmdline
UUID=df5f524b-9e81-463b-946b-fcf9034f7cb3 log_buf_len=1M net.ifnames=0 fsck.mode=force fsck.repair=yes ipv6.disable=1 loglevel=7 apparmor=0 selinux=0 cgroup_no_v1=all

If a remove lxc.cgroup2.memory.xxxxxxxx, then all containers can boot correctly, therefore

ansible@hn-rock64-130:~$ ls -lh /sys/fs/cgroup/
total 0
-r–r–r-- 1 root root 0 Mar 15 20:58 cgroup.controllers
-rw-r–r-- 1 root root 0 Mar 15 22:16 cgroup.max.depth
-rw-r–r-- 1 root root 0 Mar 15 22:16 cgroup.max.descendants
-rw-r–r-- 1 root root 0 Mar 15 22:16 cgroup.pressure
-rw-r–r-- 1 root root 0 Mar 15 22:16 cgroup.procs
-r–r–r-- 1 root root 0 Mar 15 22:16 cgroup.stat
-rw-r–r-- 1 root root 0 Mar 15 20:58 cgroup.subtree_control
-rw-r–r-- 1 root root 0 Mar 15 20:58 cgroup.threads
-rw-r–r-- 1 root root 0 Mar 15 22:16 cpu.pressure
-r–r–r-- 1 root root 0 Mar 15 22:16 cpu.stat
-r–r–r-- 1 root root 0 Mar 15 22:16 cpu.stat.local
-r–r–r-- 1 root root 0 Mar 15 22:16 cpuset.cpus.effective
-r–r–r-- 1 root root 0 Mar 15 22:16 cpuset.cpus.isolated
-r–r–r-- 1 root root 0 Mar 15 22:16 cpuset.mems.effective
-rw-r–r-- 1 root root 0 Mar 15 22:16 io.cost.model
-rw-r–r-- 1 root root 0 Mar 15 22:16 io.cost.qos
-rw-r–r-- 1 root root 0 Mar 15 22:16 io.pressure
-r–r–r-- 1 root root 0 Mar 15 22:16 io.stat
drwxr-xr-x 2 root root 0 Mar 15 21:02 lxc.monitor.vn-bookworm-arm64-130
drwxr-xr-x 2 root root 0 Mar 15 21:02 lxc.monitor.vn-bullseye-arm64-130
drwxr-xr-x 2 root root 0 Mar 15 21:02 lxc.monitor.vn-buster-arm64-130
drwxr-xr-x 2 root root 0 Mar 15 21:02 lxc.monitor.vn-debian-arm64-130
drwxr-xr-x 2 root root 0 Mar 15 21:02 lxc.monitor.vn-ntp-130
drwxr-xr-x 2 root root 0 Mar 15 21:02 lxc.payload.vn-bookworm-arm64-130
drwxr-xr-x 2 root root 0 Mar 15 21:02 lxc.payload.vn-bullseye-arm64-130
drwxr-xr-x 2 root root 0 Mar 15 21:02 lxc.payload.vn-buster-arm64-130
drwxr-xr-x 2 root root 0 Mar 15 21:02 lxc.payload.vn-debian-arm64-130
drwxr-xr-x 2 root root 0 Mar 15 21:02 lxc.payload.vn-ntp-130
drwxr-xr-x 2 root root 0 Mar 15 20:58 lxc.pivot
-r–r–r-- 1 root root 0 Mar 15 22:16 memory.numa_stat
-rw-r–r-- 1 root root 0 Mar 15 22:16 memory.pressure
–w------- 1 root root 0 Mar 15 20:58 memory.reclaim
-r–r–r-- 1 root root 0 Mar 15 22:16 memory.stat
-rw-r–r-- 1 root root 0 Mar 15 22:16 memory.zswap.writeback
-r–r–r-- 1 root root 0 Mar 15 22:16 misc.capacity
-r–r–r-- 1 root root 0 Mar 15 22:16 misc.current
-r–r–r-- 1 root root 0 Mar 15 22:16 misc.peak

stgraber · March 15, 2025, 11:37pm

Can you show ls -lh /sys/fs/cgroup/lxc.payload.vn-ntp-130?

stgraber · March 15, 2025, 11:39pm

Actually, I suspect the issue is quite simple.

You’re using cgroup2 which doesn’t have limit_in_bytes and memsw.limit_in_bytes but instead uses memory.max and memory.swap.max`

jean-marc-LACROIX · March 17, 2025, 9:52pm

Thanks Stephane for this suggestion. I’ll be testing this soon. Question? Where can I find the exact list of LXC parameters in cgroupv2 mode? Thanks in advance for an up-to-date link.

stgraber · March 17, 2025, 9:56pm

It’s based on filesystem paths.

lxc.cgroup.CONTROLLER.FILE = VALUE for cgroup1 (/sys/fs/cgroup/CONTROLLER/CONTROLLER.FILE)
lxc.cgroup2.CONTROLLER.FILE = VALUE for cgroup2 (/sys/fs/cgroup/CONTROLLER.FILE)