There is no useable cpuset controller - Failed to setup limits for the "cpuset" controller. The controller seems to be unused by "cgfsng" cgroup driver or not enabled on the cgroup hierarchy

LXC
1

Hi,

debian 10 host

lxc-start results in:

lxc-start ubuntu 20190917221541.721 WARN     cgfsng - cgroups/cgfsng.c:get_hierarchy:204 - There is no useable cpuset controller
lxc-start ubuntu 20190917221541.721 ERROR    cgfsng - cgroups/cgfsng.c:cg_legacy_set_data:2191 - Failed to setup limits for the "cpuset" controller. The controller seems to be unused by "cgfsng" cgroup driver or not enabled on the cgroup hierarchy

I punished the searchengines now for quiet a while, but there seems not to be an information available, how to setup / activate that cpuset controller.

It would be great, if someone could redeem me.

Thank you!

$ cat /proc/self/cgroup 
11:memory:/user/lxc/0
10:devices:/user.slice
9:perf_event:/
8:pids:/user.slice/user-0.slice/session-1.scope
7:cpu,cpuacct:/user.slice
6:rdma:/
5:freezer:/user/lxc/0
4:cpuset:/
3:blkio:/user.slice
2:net_cls,net_prio:/
1:name=systemd:/user/lxc/0
0::/user.slice/user-0.slice/session-1.scope

$ lxc-checkconfig

Kernel configuration not found at /proc/config.gz; searching...
Kernel configuration found at /boot/config-4.19.0-6-amd64
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled

--- Control groups ---
Cgroups: enabled

Cgroup v1 mount points: 
/sys/fs/cgroup/systemd
/sys/fs/cgroup/net_cls,net_prio
/sys/fs/cgroup/blkio
/sys/fs/cgroup/cpuset
/sys/fs/cgroup/freezer
/sys/fs/cgroup/rdma
/sys/fs/cgroup/cpu,cpuacct
/sys/fs/cgroup/pids
/sys/fs/cgroup/perf_event
/sys/fs/cgroup/devices
/sys/fs/cgroup/memory

Cgroup v2 mount points: 
/sys/fs/cgroup/unified

Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, not loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, not loaded
Advanced netfilter: enabled, loaded
CONFIG_NF_NAT_IPV4: enabled, loaded
CONFIG_NF_NAT_IPV6: enabled, loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, loaded

--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities: 

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

$ mount

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=16064776k,nr_inodes=4016194,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3216304k,mode=755)
/dev/vda2 on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (rw,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset,clone_children)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16023)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3216300k,mode=700)

container config:

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template:
# Template script checksum (SHA-1): 273c51343604eb85f7e294c8da0a5eb769d648f3
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)

# "Secure" mounting

# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf

# For Ubuntu 14.04
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none bind,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none bind,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none bind,optional 0 0
lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir,optional 0 0
lxc.include = /usr/share/lxc/config/userns.conf

# For Ubuntu 14.04
lxc.mount.entry = /sys/firmware/efi/efivars sys/firmware/efi/efivars none bind,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none bind,optional 0 0
lxc.arch = linux64

lxc.apparmor.profile = unconfined


# Container specific configuration
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
lxc.mount.auto = proc:mixed sys:ro cgroup:mixed
/var/lib/lxc/ = ~/.local/share/lxc
/var/cache/lxc = ~/.cache/lxc
lxc.rootfs.path = dir:/home/lxc/.local/share/lxc/ubuntu/rootfs
lxc.uts.name = ubuntu

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = ovsbr
lxc.net.0.flags = up
lxc.net.0.name = eth0
lxc.net.0.hwaddr = 00:16:3e:48:cd:59

lxc.cgroup.cpuset.cpus = 1
lxc.cgroup.memory.limit_in_bytes = 2000000000
2

4:cpuset:/ this means that your user doesn’t have a dedicated path in the cpuset controller, so nowhere your user can write, causing the error.

3

Hi Stéphane,

yes i already thought so too.

My problem is, that i can find exactly 0 documentation about howto change that exactly.

All in found is, that you have to install cgfs or what so ever and it will handle it automatically for you.

But an information how exactly to give a user this rights and howto create this dedicated path in the cgroups to make it possible for unprivileged users to access also cpuset or device or what ever is missing, i was not able to find so far.

Could you please point me to some information about that or kindly tell me how to configure this ?

It seems there were quiet some changes between lxc v2 and v3, so somehow the internet is full of outdated informations or handle about lxd that does not seem to know about this problems anyway.

So i am really totally lost and stuck here and would be very thankful if you could help me out with this.

Thank you!

4

You’d need to create a directory structure under /sys/fs/cgroup/cpuset similar to that systemd created for you on the other controllers. Then change ownership of that path to your user and finally, move your shell into that cgroup by writing its pid to the tasks file inside the cgroup.

5

Hi,

thank you very much! I will follow your orders.

I was wondering if this is the normal workflow. Isnt there some service available which would usually make sure, that all controllers are loaded for the users ?

Of course i can every time, if want to start / create an unprivileged lxc manually write the pid of my login shell into the /sys tree to the correct position. And of course i can also write a script, that, as soon as i login, will do that automatically.

But maybe there is a more elegant way, some kind of service you can name that will / would normally do that for me. In that case i could check what this service is (not) doing on this system, fix it ( hopefully ) and write here a little howto/walk through for it.

6

Guys, i’m no expert on these but seems you guys can help me-
i’m trying to join intel debian nuc to as worker node to my kubernetes cluster and facing CPUset missing.

Any help please.

more /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
CPUset 0 182 1

more /proc/self/cgroups

in this file i have nothing related to cpu or Cgroups

7

Either your host system doesn’t have any of the cgroup hierarchies mounted or it’s using the unified hierarchy (which would show as a 0: line in /proc/self/cgroup).