Failed creating cgroups

LXC
#1

I’m on debian testing, my containers used to start without issue, last time I started them was on 18th of may :slight_smile: A long time agop I know.

Today for every container I have, even newly created one I got this cgroups issue.

  lxc-start 20170707075123.911 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/freezer/
  lxc-start 20170707075123.911 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/pids/user.slice/user-1000.slice/session-2.scope
  lxc-start 20170707075123.911 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/devices/user.slice
  lxc-start 20170707075123.911 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpu,cpuacct/user.slice
  lxc-start 20170707075123.911 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/net_cls,net_prio/
  lxc-start 20170707075123.911 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpuset/
  lxc-start 20170707075123.912 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/memory/user.slice
  lxc-start 20170707075123.912 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/systemd/user.slice/user-1000.slice/session-2.scope
  lxc-start 20170707075123.912 ERROR    lxc_start - start.c:lxc_spawn:1119 - Failed creating cgroups.
  lxc-start 20170707075123.912 ERROR    lxc_start - start.c:__lxc_start:1354 - Failed to spawn container "modoboa".
  lxc-start 20170707075129.450 ERROR    lxc_start_ui - tools/lxc_start.c:main:366 - The container failed to start.

➜ ~ cat /proc/self/cgroup

10:blkio:/user.slice
9:perf_event:/
8:freezer:/
7:pids:/user.slice/user-1000.slice/session-2.scope
6:devices:/user.slice
5:cpu,cpuacct:/user.slice
4:net_cls,net_prio:/
3:cpuset:/
2:memory:/user.slice
1:name=systemd:/user.slice/user-1000.slice/session-2.scope
0::/user.slice/user-1000.slice/session-2.scope

I change nothing in my config, just updated some packages those last 2 months so I can;t say it’s after such or such upgrade. Any clue where I could look at ?

O and looking a little root owns it

➜  ~ ls -la /sys/fs/cgroup/systemd/user.slice/user-1000.slice/
total 0
drwxr-xr-x  4 root  root  0 Jul  7 10:13 .
drwxr-xr-x  3 root  root  0 Jul  7 09:50 ..
-rw-r--r--  1 root  root  0 Jul  7 10:13 cgroup.clone_children
-rw-r--r--  1 root  root  0 Jul  7 10:13 cgroup.procs
-rw-r--r--  1 root  root  0 Jul  7 10:13 notify_on_release
drwxr-xr-x  2 root  root  0 Jul  7 10:13 session-2.scope
-rw-r--r--  1 root  root  0 Jul  7 10:13 tasks
drwxr-xr-x 31 lotso lotso 0 Jul  7 09:51 user@1000.service

but below my user has that :

➜  ~ ls -la /sys/fs/cgroup/systemd/user.slice/user-1000.slice/user@1000.service/var-lib-lxcfs.mount/
total 0
drwxr-xr-x  2 lotso lotso 0 Jul  7 09:51 .
drwxr-xr-x 31 lotso lotso 0 Jul  7 09:51 ..
-rw-r--r--  1 lotso lotso 0 Jul  7 09:51 cgroup.clone_children
-rw-r--r--  1 lotso lotso 0 Jul  7 09:51 cgroup.procs
-rw-r--r--  1 lotso lotso 0 Jul  7 09:51 notify_on_release
-rw-r--r--  1 lotso lotso 0 Jul  7 09:51 tasks
#2

Can you please post the LXC version you’re using.

#3

Posting a full debug log would be useful too. Pass “-o debug -l trace” to your lxc-start, then post the content of the “debug” file.

#4

Some more info

The lxc version I use

➜  ~ sudo aptitude show lxc
[sudo] password for lotso:
Package: lxc
Version: 1:2.0.8-1
State: installed
Automatically installed: no

the debug

  lxc-start 20170707164658.858 INFO     lxc_start_ui - tools/lxc_start.c:main:275 - using rcfile /home/lotso/.local/share/lxc/modoboa/config
  lxc-start 20170707164658.858 WARN     lxc_confile - confile.c:config_pivotdir:1916 - lxc.pivotdir is ignored.  It will soon become an error.
  lxc-start 20170707164658.858 INFO     lxc_confile - confile.c:config_idmap:1537 - read uid map: type u nsid 0 hostid 1279648 range 65536
  lxc-start 20170707164658.858 INFO     lxc_confile - confile.c:config_idmap:1537 - read uid map: type g nsid 0 hostid 1279648 range 65536
  lxc-start 20170707164658.858 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:330 - Going to wait for pid 9661.
  lxc-start 20170707164658.859 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:349 - Trying to sync with child process.
  lxc-start 20170707164658.859 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 3.
  lxc-start 20170707164658.859 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 4.
  lxc-start 20170707164658.859 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 5.
  lxc-start 20170707164658.859 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 6.
  lxc-start 20170707164658.859 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 7.
  lxc-start 20170707164658.859 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 9.
  lxc-start 20170707164658.859 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 30.
  lxc-start 20170707164658.859 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 33.
  lxc-start 20170707164658.859 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 34.
  lxc-start 20170707164658.859 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:387 - Using pipe file descriptor 10 for monitord.
  lxc-start 20170707164658.862 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:364 - Successfully synced with child process.
  lxc-start 20170707164658.863 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:333 - Finished waiting on pid 9661.
  lxc-start 20170707164658.863 INFO     lxc_container - lxccontainer.c:do_lxcapi_start:802 - Attempting to set proc title to [lxc monitor] /home/lotso/.local/share/lxc modoboa
  lxc-start 20170707164658.863 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 3.
  lxc-start 20170707164658.863 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 4.
  lxc-start 20170707164658.863 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 5.
  lxc-start 20170707164658.863 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 6.
  lxc-start 20170707164658.863 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 7.
  lxc-start 20170707164658.863 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 30.
  lxc-start 20170707164658.863 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 33.
  lxc-start 20170707164658.863 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 34.
  lxc-start 20170707164658.863 INFO     lxc_lsm - lsm/lsm.c:lsm_init:48 - LSM security driver nop
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:418 - processing: .reject_force_umount  # comment this to allow umount -f;  not recommended.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:590 - Adding native rule for reject_force_umount action 0.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:259 - Setting Seccomp rule to reject force umounts.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:593 - Adding compat rule for reject_force_umount action 0.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:259 - Setting Seccomp rule to reject force umounts.
  lxc-start 20170707164658.864 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:330 - Going to wait for pid 9665.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:259 - Setting Seccomp rule to reject force umounts.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:418 - processing: .[all].
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:418 - processing: .kexec_load errno 1.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:590 - Adding native rule for kexec_load action 327681.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:593 - Adding compat rule for kexec_load action 327681.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:418 - processing: .open_by_handle_at errno 1.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:590 - Adding native rule for open_by_handle_at action 327681.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:593 - Adding compat rule for open_by_handle_at action 327681.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:418 - processing: .init_module errno 1.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:590 - Adding native rule for init_module action 327681.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:593 - Adding compat rule for init_module action 327681.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:418 - processing: .finit_module errno 1.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:590 - Adding native rule for finit_module action 327681.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:593 - Adding compat rule for finit_module action 327681.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:418 - processing: .delete_module errno 1.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:590 - Adding native rule for delete_module action 327681.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:593 - Adding compat rule for delete_module action 327681.
  lxc-start 20170707164658.864 INFO     lxc_seccomp - seccomp.c:parse_config_v2:603 - Merging in the compat Seccomp ctx into the main one.
  lxc-start 20170707164658.864 DEBUG    lxc_start - start.c:setup_signal_fd:273 - Set SIGCHLD handler with file descriptor: 4.
  lxc-start 20170707164658.864 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:349 - Trying to sync with child process.
  lxc-start 20170707164658.864 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 3.
  lxc-start 20170707164658.864 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 4.
  lxc-start 20170707164658.864 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 5.
  lxc-start 20170707164658.864 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 6.
  lxc-start 20170707164658.864 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 7.
  lxc-start 20170707164658.864 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 9.
  lxc-start 20170707164658.864 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 30.
  lxc-start 20170707164658.864 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 33.
  lxc-start 20170707164658.864 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 34.
  lxc-start 20170707164658.864 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:387 - Using pipe file descriptor 10 for monitord.
  lxc-start 20170707164658.864 DEBUG    console - console.c:lxc_console_peer_default:438 - process does not have a controlling terminal
  lxc-start 20170707164658.868 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:364 - Successfully synced with child process.
  lxc-start 20170707164658.868 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:333 - Finished waiting on pid 9665.
  lxc-start 20170707164658.868 INFO     lxc_monitor - monitor.c:lxc_monitor_sock_name:201 - using monitor socket name "lxc/6a52fbbcf8911aff//home/lotso/.local/share/lxc" (length of socket name 49 must be <= 105)
  lxc-start 20170707164658.868 DEBUG    lxc_monitor - monitor.c:lxc_monitor_open:225 - opening monitor socket lxc/6a52fbbcf8911aff//home/lotso/.local/share/lxc with len 49
  lxc-start 20170707164658.914 INFO     lxc_start - start.c:lxc_init:475 - Container "modoboa" is initialized.
  lxc-start 20170707164658.915 DEBUG    lxc_start - start.c:__lxc_start:1325 - Not dropping CAP_SYS_BOOT or watching utmp.
  lxc-start 20170707164658.915 INFO     lxc_cgroup - cgroups/cgroup.c:cgroup_init:68 - cgroup driver cgroupfs initing for modoboa
  lxc-start 20170707164658.915 ERROR    lxc_cgfs - cgroups/cgfs.c:lxc_cgroupfs_create:909 - Could not set clone_children to 1 for cpuset hierarchy in parent cgroup.
  lxc-start 20170707164658.915 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/blkio/user.slice
  lxc-start 20170707164658.915 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/perf_event/
  lxc-start 20170707164658.915 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/freezer/
  lxc-start 20170707164658.915 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/pids/user.slice/user-1000.slice/session-138.scope
  lxc-start 20170707164658.915 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/devices/user.slice
  lxc-start 20170707164658.915 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpu,cpuacct/user.slice
  lxc-start 20170707164658.915 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/net_cls,net_prio/
  lxc-start 20170707164658.915 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpuset/
  lxc-start 20170707164658.915 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/memory/user.slice
  lxc-start 20170707164658.915 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/systemd/user.slice/user-1000.slice/session-138.scope
  lxc-start 20170707164658.915 ERROR    lxc_start - start.c:lxc_spawn:1119 - Failed creating cgroups.
  lxc-start 20170707164658.915 ERROR    lxc_start - start.c:__lxc_start:1354 - Failed to spawn container "modoboa".
  lxc-start 20170707164658.915 INFO     lxc_conf - conf.c:run_script_argv:427 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "modoboa", config section "lxc".
  lxc-start 20170707164659.420 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:177 - Command get_cgroup failed to receive response: Connection reset by peer.
  lxc-start 20170707164704.442 ERROR    lxc_start_ui - tools/lxc_start.c:main:366 - The container failed to start.
  lxc-start 20170707164704.442 ERROR    lxc_start_ui - tools/lxc_start.c:main:368 - To get more details, run the container in foreground mode.
  lxc-start 20170707164704.442 ERROR    lxc_start_ui - tools/lxc_start.c:main:370 - Additional information can be obtained by setting the --logfile and --logpriority options.

this particular config

➜  ~ cat /home/lotso/.local/share/lxc/modoboa/config
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --dist ubuntu --release xenial --arch amd64
# Template script checksum (SHA-1): 740c51206e35463362b735e68b867876048a8baf
# For additional config options, please look at lxc.container.conf(5)
# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)
# Subuids and subgids mapping
# "Secure" mounting
# Network configuration
# Distribution configuration
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.include = /usr/share/lxc/config/ubuntu.userns.conf
lxc.arch = x86_64
# Container specific configuration
lxc.include = /etc/lxc/default.conf
lxc.id_map = u 0 1279648 65536
lxc.id_map = g 0 1279648 65536
lxc.mount.auto = proc:mixed sys:ro cgroup:mixed
lxc.rootfs = /home/lotso/.local/share/lxc/modoboa/rootfs
lxc.rootfs.backend = dir
lxc.utsname = modoboa
# Network configuration
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:FF:2a:a8:38:3b

the default config for unprivileged

➜  ~ cat .config/lxc/default.conf
lxc.include = /etc/lxc/default.conf
# Subuids and subgids mapping
lxc.id_map = u 0 1279648 65536
lxc.id_map = g 0 1279648 65536
# "Secure" mounting
lxc.mount.auto = proc:mixed sys:ro cgroup:mixed


# Network configuration
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:FF:xx:xx:xx:xx
#5

Thanks. Looks like LXC is unhappy with your cgroup ownership which is somewhat confirmed by the original listing. LXC expects you to at least own your path in the freezer cgroup.

The easiest way to have this done for you is to install the libpam-cgfs package which will then automatically setup cgroup ownership for you at login time.

#6

libpam-cgfs is installed , I tried reinstalling it , logout, login, seems like it didn’t change anything.

what’s weird is that this was working 2 months ago, I’ll try a check if those packages changed, later

#7

Oh, I bet I know what the problem is.

We fixed a couple of issues with libpam-cgfs after the lxcfs 2.0.7 release. Those have been cherry-picked into the Ubuntu package but I suspect the Debian one is lacking those.

Debian and Ubuntu should be pretty much binary compatible these days, so you could try to download https://launchpad.net/ubuntu/+source/lxcfs/2.0.7-0ubuntu1~17.04.2/+build/12855295/+files/libpam-cgfs_2.0.7-0ubuntu1~17.04.2_amd64.deb and install it on your system. That’s the patched version we have in Ubuntu right now which includes the bits needed for systems using the unified cgroup hierarchy (which is likely what changed for you after a systemd upgrade).

#8

I uninstalled the debian package, installed the ubuntu one you linked, logged out, logged in, I got the same effect.
I tried a reboot too.
Systemd playing bad on me ?

#9

Nope, it’s just me giving you the wrong package… The one I gave you doesn’t have the right cherry-picks…

#10

Try https://launchpad.net/ubuntu/+source/lxcfs/2.0.7-0ubuntu4/+build/12785691/+files/libpam-cgfs_2.0.7-0ubuntu4_amd64.deb instead

#11

➜ ~ lxc-start -n modoboa -o debug -l trace
➜ ~ lxc-ls -f
NAME STATE AUTOSTART GROUPS IPV4 IPV6
modoboa RUNNING 0 - 10.0.3.191 -

Success ! thanks for your patience, will report that to debian :slight_smile:

#12

Glad that it worked!

LXCFS 2.0.8 will have the fix for sure, but it’d be great if Debian could cherry-pick it until then.

Unable to start LXC unprivileged container: ERROR lxc_cgfs - cgroups Permission denied
#13

Hello @stgraber , I have exactly the same problem with cgroups but unfortunately your suggested Ubuntu cherry pick .deb is not available for my armhf hardware.
Do you know where I can get it from?

I am running Debian Stretch on armhf hardware with Armbian OS.
LXC 2.0.7
libpam-cgfs/oldstable,now 2.0.7-1+deb9u1 armhf

Thanky you :slight_smile: