Problem with resources limitation (cpu, memory, ...)

I installed debian 11 on a raspberry pi 4 (4GB of RAM) from the latest image available here: Tested images

I installed LXC and LXCFS from repositories but the resources limitation don’t seem to work properly. A container limited to one CPU can use all 4 CPUs (tested with the stress-ng command in the container).

I can’t figure out what is wrong. The same installation worked on an older version of raspiban and LXC (32bit based on debian 10)

Here is installation information:

root@hyper:~# lxcfs --version
4.0.7

root@hyper:~# lxc-checkconfig
LXC version 4.0.6
Kernel configuration not found at /proc/config.gz; searching...
Kernel configuration found at /boot/config-5.10.0-11-arm64
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled

--- Control groups ---
Cgroups: enabled

Cgroup v1 mount points:


Cgroup v2 mount points:
/sys/fs/cgroup

Cgroup v1 systemd controller: missing
Cgroup v1 freezer controller: missing
Cgroup namespace: required
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, loaded
CONFIG_NF_NAT_IPV4: missing
CONFIG_NF_NAT_IPV6: missing
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, loaded

--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities:

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

root@hyper:~# systemctl status lxcfs.service
● lxcfs.service - FUSE filesystem for LXC
     Loaded: loaded (/lib/systemd/system/lxcfs.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-02-19 23:25:39 UTC; 8h ago
       Docs: man:lxcfs(1)
   Main PID: 530 (lxcfs)
      Tasks: 3 (limit: 4429)
     Memory: 1.0M
        CPU: 37ms
     CGroup: /system.slice/lxcfs.service
             └─530 /usr/bin/lxcfs /var/lib/lxcfs

févr. 19 23:25:39 hyper lxcfs[530]: - proc_diskstats
févr. 19 23:25:39 hyper lxcfs[530]: - proc_loadavg
févr. 19 23:25:39 hyper lxcfs[530]: - proc_meminfo
févr. 19 23:25:39 hyper lxcfs[530]: - proc_stat
févr. 19 23:25:39 hyper lxcfs[530]: - proc_swaps
févr. 19 23:25:39 hyper lxcfs[530]: - proc_uptime
févr. 19 23:25:39 hyper lxcfs[530]: - shared_pidns
févr. 19 23:25:39 hyper lxcfs[530]: - cpuview_daemon
févr. 19 23:25:39 hyper lxcfs[530]: - loadavg_daemon
févr. 19 23:25:39 hyper lxcfs[530]: - pidfds

root@hyper:~# ls -l /var/lib/lxcfs/
ls: impossible d'accéder à '/var/lib/lxcfs/cgroup': Erreur d'entrée/sortie
total 0
?????????? ? ?    ?    ?              ? cgroup
dr-xr-xr-x 2 root root 0 20 févr. 08:16 proc
dr-xr-xr-x 2 root root 0 20 févr. 08:16 sys

root@hyper:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye

Here is the configuration of the container:

root@hyper:~# cat /var/lib/lxc/dev/config
# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.include = /usr/share/lxc/config/userns.conf
lxc.arch = linux64

# Container specific configuration
lxc.apparmor.allow_nesting = 1
lxc.apparmor.profile = generated
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
lxc.rootfs.path = lvm:/dev/lxc/dev
lxc.uts.name = dev

# Network configuration
lxc.net.0.name = eth0
lxc.net.0.type = veth
lxc.net.0.link = br0
lxc.net.0.flags = up
lxc.net.0.hwaddr = FC:FC:48:F3:B6:3C
# Resources configuration
lxc.cgroup.memory.limit_in_bytes = 64M
lxc.cgroup.cpuset.cpus = 1
lxc.cgroup.cpu.shares = 10
# Boot configuration
lxc.start.auto = 0
lxc.start.delay = 10
lxc.start.order = 1

root@hyper:~# cat /sys/fs/cgroup/lxc.payload.dev/cpuset.cpus

root@hyper:~#

Some info from the container:

~ # grep processor /proc/cpuinfo
processor   : 0
processor   : 1
processor   : 2
processor   : 3

~ # grep MemTotal /proc/meminfo
MemTotal:        3881588 kB

 # cat /proc/mounts
/dev/lxc/dev / ext4 rw,relatime,stripe=64 0 0
none /dev tmpfs rw,relatime,size=492k,mode=755,uid=100000,gid=100000 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
proc /proc/sys/net proc rw,nosuid,nodev,noexec,relatime 0 0
proc /proc/sys proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/sysrq-trigger proc ro,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
proc /dev/.lxc/proc proc rw,relatime 0 0
sys /dev/.lxc/sys sysfs rw,relatime 0 0
cgroup /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime 0 0
udev /dev/full devtmpfs rw,nosuid,relatime,size=1890016k,nr_inodes=472504,mode=755 0 0
udev /dev/null devtmpfs rw,nosuid,relatime,size=1890016k,nr_inodes=472504,mode=755 0 0
udev /dev/random devtmpfs rw,nosuid,relatime,size=1890016k,nr_inodes=472504,mode=755 0 0
udev /dev/tty devtmpfs rw,nosuid,relatime,size=1890016k,nr_inodes=472504,mode=755 0 0
udev /dev/urandom devtmpfs rw,nosuid,relatime,size=1890016k,nr_inodes=472504,mode=755 0 0
udev /dev/zero devtmpfs rw,nosuid,relatime,size=1890016k,nr_inodes=472504,mode=755 0 0
devpts /dev/console devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
none /proc/sys/kernel/random/boot_id tmpfs ro,nosuid,nodev,noexec,relatime,size=492k,mode=755,uid=100000,gid=100000 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1024 0 0
devpts /dev/ptmx devpts rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1024 0 0
devpts /dev/tty1 devpts rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1024 0 0
devpts /dev/tty2 devpts rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1024 0 0
devpts /dev/tty3 devpts rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1024 0 0
devpts /dev/tty4 devpts rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1024 0 0
tmpfs /run tmpfs rw,nosuid,nodev,size=776320k,nr_inodes=819200,mode=755,uid=100000,gid=100000 0

Does anyone know what the problem is?

Hi,

I can’t reproduce this under the official Debian 11 image running on amd64. Maybe this is something related to the arm64 version and/or specific rpi hacks?

I don’t think the problem is related to a rpi hack. With the debian 11 image there is no hack others than boot sequence before loading the initramfs.

With cgroupv1 enabled in hybrid mode (added systemd.unified_cgroup_hierarchy=false systemd.legacy_systemd_cgroup_controller=false parameters to the kernel at boot) everything works fine.
I don’t know where the problem is located.

Edit: just found the solution. lxc keys name differ from cgroupv1, explained here : Limitations not applied inside of containers