euri10
July 7, 2017, 8:15am
1
I’m on debian testing, my containers used to start without issue, last time I started them was on 18th of may
Today for every container I have, even newly created one I got this cgroups issue.
lxc-start 20170707075123.911 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/freezer/
lxc-start 20170707075123.911 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/pids/user.slice/user-1000.slice/session-2.scope
lxc-start 20170707075123.911 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/devices/user.slice
lxc-start 20170707075123.911 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpu,cpuacct/user.slice
lxc-start 20170707075123.911 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/net_cls,net_prio/
lxc-start 20170707075123.911 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpuset/
lxc-start 20170707075123.912 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/memory/user.slice
lxc-start 20170707075123.912 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/systemd/user.slice/user-1000.slice/session-2.scope
lxc-start 20170707075123.912 ERROR lxc_start - start.c:lxc_spawn:1119 - Failed creating cgroups.
lxc-start 20170707075123.912 ERROR lxc_start - start.c:__lxc_start:1354 - Failed to spawn container "modoboa".
lxc-start 20170707075129.450 ERROR lxc_start_ui - tools/lxc_start.c:main:366 - The container failed to start.
➜ ~ cat /proc/self/cgroup
10:blkio:/user.slice
9:perf_event:/
8:freezer:/
7:pids:/user.slice/user-1000.slice/session-2.scope
6:devices:/user.slice
5:cpu,cpuacct:/user.slice
4:net_cls,net_prio:/
3:cpuset:/
2:memory:/user.slice
1:name=systemd:/user.slice/user-1000.slice/session-2.scope
0::/user.slice/user-1000.slice/session-2.scope
I change nothing in my config, just updated some packages those last 2 months so I can;t say it’s after such or such upgrade. Any clue where I could look at ?
O and looking a little root owns it
➜ ~ ls -la /sys/fs/cgroup/systemd/user.slice/user-1000.slice/
total 0
drwxr-xr-x 4 root root 0 Jul 7 10:13 .
drwxr-xr-x 3 root root 0 Jul 7 09:50 ..
-rw-r--r-- 1 root root 0 Jul 7 10:13 cgroup.clone_children
-rw-r--r-- 1 root root 0 Jul 7 10:13 cgroup.procs
-rw-r--r-- 1 root root 0 Jul 7 10:13 notify_on_release
drwxr-xr-x 2 root root 0 Jul 7 10:13 session-2.scope
-rw-r--r-- 1 root root 0 Jul 7 10:13 tasks
drwxr-xr-x 31 lotso lotso 0 Jul 7 09:51 user@1000.service
but below my user has that :
➜ ~ ls -la /sys/fs/cgroup/systemd/user.slice/user-1000.slice/user@1000.service/var-lib-lxcfs.mount/
total 0
drwxr-xr-x 2 lotso lotso 0 Jul 7 09:51 .
drwxr-xr-x 31 lotso lotso 0 Jul 7 09:51 ..
-rw-r--r-- 1 lotso lotso 0 Jul 7 09:51 cgroup.clone_children
-rw-r--r-- 1 lotso lotso 0 Jul 7 09:51 cgroup.procs
-rw-r--r-- 1 lotso lotso 0 Jul 7 09:51 notify_on_release
-rw-r--r-- 1 lotso lotso 0 Jul 7 09:51 tasks
brauner
July 7, 2017, 2:47pm
2
Can you please post the LXC version you’re using.
stgraber
July 7, 2017, 4:10pm
3
Posting a full debug log would be useful too. Pass “-o debug -l trace” to your lxc-start, then post the content of the “debug” file.
euri10
July 7, 2017, 4:49pm
4
Some more info
The lxc version I use
➜ ~ sudo aptitude show lxc
[sudo] password for lotso:
Package: lxc
Version: 1:2.0.8-1
State: installed
Automatically installed: no
the debug
lxc-start 20170707164658.858 INFO lxc_start_ui - tools/lxc_start.c:main:275 - using rcfile /home/lotso/.local/share/lxc/modoboa/config
lxc-start 20170707164658.858 WARN lxc_confile - confile.c:config_pivotdir:1916 - lxc.pivotdir is ignored. It will soon become an error.
lxc-start 20170707164658.858 INFO lxc_confile - confile.c:config_idmap:1537 - read uid map: type u nsid 0 hostid 1279648 range 65536
lxc-start 20170707164658.858 INFO lxc_confile - confile.c:config_idmap:1537 - read uid map: type g nsid 0 hostid 1279648 range 65536
lxc-start 20170707164658.858 DEBUG lxc_monitor - monitor.c:lxc_monitord_spawn:330 - Going to wait for pid 9661.
lxc-start 20170707164658.859 DEBUG lxc_monitor - monitor.c:lxc_monitord_spawn:349 - Trying to sync with child process.
lxc-start 20170707164658.859 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 3.
lxc-start 20170707164658.859 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 4.
lxc-start 20170707164658.859 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 5.
lxc-start 20170707164658.859 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 6.
lxc-start 20170707164658.859 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 7.
lxc-start 20170707164658.859 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 9.
lxc-start 20170707164658.859 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 30.
lxc-start 20170707164658.859 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 33.
lxc-start 20170707164658.859 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 34.
lxc-start 20170707164658.859 DEBUG lxc_monitor - monitor.c:lxc_monitord_spawn:387 - Using pipe file descriptor 10 for monitord.
lxc-start 20170707164658.862 DEBUG lxc_monitor - monitor.c:lxc_monitord_spawn:364 - Successfully synced with child process.
lxc-start 20170707164658.863 DEBUG lxc_monitor - monitor.c:lxc_monitord_spawn:333 - Finished waiting on pid 9661.
lxc-start 20170707164658.863 INFO lxc_container - lxccontainer.c:do_lxcapi_start:802 - Attempting to set proc title to [lxc monitor] /home/lotso/.local/share/lxc modoboa
lxc-start 20170707164658.863 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 3.
lxc-start 20170707164658.863 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 4.
lxc-start 20170707164658.863 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 5.
lxc-start 20170707164658.863 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 6.
lxc-start 20170707164658.863 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 7.
lxc-start 20170707164658.863 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 30.
lxc-start 20170707164658.863 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 33.
lxc-start 20170707164658.863 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 34.
lxc-start 20170707164658.863 INFO lxc_lsm - lsm/lsm.c:lsm_init:48 - LSM security driver nop
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:418 - processing: .reject_force_umount # comment this to allow umount -f; not recommended.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:590 - Adding native rule for reject_force_umount action 0.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:do_resolve_add_rule:259 - Setting Seccomp rule to reject force umounts.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:593 - Adding compat rule for reject_force_umount action 0.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:do_resolve_add_rule:259 - Setting Seccomp rule to reject force umounts.
lxc-start 20170707164658.864 DEBUG lxc_monitor - monitor.c:lxc_monitord_spawn:330 - Going to wait for pid 9665.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:do_resolve_add_rule:259 - Setting Seccomp rule to reject force umounts.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:418 - processing: .[all].
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:418 - processing: .kexec_load errno 1.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:590 - Adding native rule for kexec_load action 327681.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:593 - Adding compat rule for kexec_load action 327681.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:418 - processing: .open_by_handle_at errno 1.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:590 - Adding native rule for open_by_handle_at action 327681.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:593 - Adding compat rule for open_by_handle_at action 327681.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:418 - processing: .init_module errno 1.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:590 - Adding native rule for init_module action 327681.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:593 - Adding compat rule for init_module action 327681.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:418 - processing: .finit_module errno 1.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:590 - Adding native rule for finit_module action 327681.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:593 - Adding compat rule for finit_module action 327681.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:418 - processing: .delete_module errno 1.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:590 - Adding native rule for delete_module action 327681.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:593 - Adding compat rule for delete_module action 327681.
lxc-start 20170707164658.864 INFO lxc_seccomp - seccomp.c:parse_config_v2:603 - Merging in the compat Seccomp ctx into the main one.
lxc-start 20170707164658.864 DEBUG lxc_start - start.c:setup_signal_fd:273 - Set SIGCHLD handler with file descriptor: 4.
lxc-start 20170707164658.864 DEBUG lxc_monitor - monitor.c:lxc_monitord_spawn:349 - Trying to sync with child process.
lxc-start 20170707164658.864 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 3.
lxc-start 20170707164658.864 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 4.
lxc-start 20170707164658.864 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 5.
lxc-start 20170707164658.864 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 6.
lxc-start 20170707164658.864 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 7.
lxc-start 20170707164658.864 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 9.
lxc-start 20170707164658.864 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 30.
lxc-start 20170707164658.864 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 33.
lxc-start 20170707164658.864 INFO lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 34.
lxc-start 20170707164658.864 DEBUG lxc_monitor - monitor.c:lxc_monitord_spawn:387 - Using pipe file descriptor 10 for monitord.
lxc-start 20170707164658.864 DEBUG console - console.c:lxc_console_peer_default:438 - process does not have a controlling terminal
lxc-start 20170707164658.868 DEBUG lxc_monitor - monitor.c:lxc_monitord_spawn:364 - Successfully synced with child process.
lxc-start 20170707164658.868 DEBUG lxc_monitor - monitor.c:lxc_monitord_spawn:333 - Finished waiting on pid 9665.
lxc-start 20170707164658.868 INFO lxc_monitor - monitor.c:lxc_monitor_sock_name:201 - using monitor socket name "lxc/6a52fbbcf8911aff//home/lotso/.local/share/lxc" (length of socket name 49 must be <= 105)
lxc-start 20170707164658.868 DEBUG lxc_monitor - monitor.c:lxc_monitor_open:225 - opening monitor socket lxc/6a52fbbcf8911aff//home/lotso/.local/share/lxc with len 49
lxc-start 20170707164658.914 INFO lxc_start - start.c:lxc_init:475 - Container "modoboa" is initialized.
lxc-start 20170707164658.915 DEBUG lxc_start - start.c:__lxc_start:1325 - Not dropping CAP_SYS_BOOT or watching utmp.
lxc-start 20170707164658.915 INFO lxc_cgroup - cgroups/cgroup.c:cgroup_init:68 - cgroup driver cgroupfs initing for modoboa
lxc-start 20170707164658.915 ERROR lxc_cgfs - cgroups/cgfs.c:lxc_cgroupfs_create:909 - Could not set clone_children to 1 for cpuset hierarchy in parent cgroup.
lxc-start 20170707164658.915 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/blkio/user.slice
lxc-start 20170707164658.915 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/perf_event/
lxc-start 20170707164658.915 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/freezer/
lxc-start 20170707164658.915 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/pids/user.slice/user-1000.slice/session-138.scope
lxc-start 20170707164658.915 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/devices/user.slice
lxc-start 20170707164658.915 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpu,cpuacct/user.slice
lxc-start 20170707164658.915 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/net_cls,net_prio/
lxc-start 20170707164658.915 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpuset/
lxc-start 20170707164658.915 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/memory/user.slice
lxc-start 20170707164658.915 ERROR lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/systemd/user.slice/user-1000.slice/session-138.scope
lxc-start 20170707164658.915 ERROR lxc_start - start.c:lxc_spawn:1119 - Failed creating cgroups.
lxc-start 20170707164658.915 ERROR lxc_start - start.c:__lxc_start:1354 - Failed to spawn container "modoboa".
lxc-start 20170707164658.915 INFO lxc_conf - conf.c:run_script_argv:427 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "modoboa", config section "lxc".
lxc-start 20170707164659.420 WARN lxc_commands - commands.c:lxc_cmd_rsp_recv:177 - Command get_cgroup failed to receive response: Connection reset by peer.
lxc-start 20170707164704.442 ERROR lxc_start_ui - tools/lxc_start.c:main:366 - The container failed to start.
lxc-start 20170707164704.442 ERROR lxc_start_ui - tools/lxc_start.c:main:368 - To get more details, run the container in foreground mode.
lxc-start 20170707164704.442 ERROR lxc_start_ui - tools/lxc_start.c:main:370 - Additional information can be obtained by setting the --logfile and --logpriority options.
this particular config
➜ ~ cat /home/lotso/.local/share/lxc/modoboa/config
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --dist ubuntu --release xenial --arch amd64
# Template script checksum (SHA-1): 740c51206e35463362b735e68b867876048a8baf
# For additional config options, please look at lxc.container.conf(5)
# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)
# Subuids and subgids mapping
# "Secure" mounting
# Network configuration
# Distribution configuration
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.include = /usr/share/lxc/config/ubuntu.userns.conf
lxc.arch = x86_64
# Container specific configuration
lxc.include = /etc/lxc/default.conf
lxc.id_map = u 0 1279648 65536
lxc.id_map = g 0 1279648 65536
lxc.mount.auto = proc:mixed sys:ro cgroup:mixed
lxc.rootfs = /home/lotso/.local/share/lxc/modoboa/rootfs
lxc.rootfs.backend = dir
lxc.utsname = modoboa
# Network configuration
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:FF:2a:a8:38:3b
the default config for unprivileged
➜ ~ cat .config/lxc/default.conf
lxc.include = /etc/lxc/default.conf
# Subuids and subgids mapping
lxc.id_map = u 0 1279648 65536
lxc.id_map = g 0 1279648 65536
# "Secure" mounting
lxc.mount.auto = proc:mixed sys:ro cgroup:mixed
# Network configuration
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:FF:xx:xx:xx:xx
stgraber
July 7, 2017, 5:20pm
5
Thanks. Looks like LXC is unhappy with your cgroup ownership which is somewhat confirmed by the original listing. LXC expects you to at least own your path in the freezer cgroup.
The easiest way to have this done for you is to install the libpam-cgfs package which will then automatically setup cgroup ownership for you at login time.
euri10
July 7, 2017, 6:26pm
6
libpam-cgfs is installed , I tried reinstalling it , logout, login, seems like it didn’t change anything.
what’s weird is that this was working 2 months ago, I’ll try a check if those packages changed, later
stgraber
July 7, 2017, 6:39pm
7
Oh, I bet I know what the problem is.
We fixed a couple of issues with libpam-cgfs after the lxcfs 2.0.7 release. Those have been cherry-picked into the Ubuntu package but I suspect the Debian one is lacking those.
Debian and Ubuntu should be pretty much binary compatible these days, so you could try to download https://launchpad.net/ubuntu/+source/lxcfs/2.0.7-0ubuntu1~17.04.2/+build/12855295/+files/libpam-cgfs_2.0.7-0ubuntu1~17.04.2_amd64.deb and install it on your system. That’s the patched version we have in Ubuntu right now which includes the bits needed for systems using the unified cgroup hierarchy (which is likely what changed for you after a systemd upgrade).
euri10
July 7, 2017, 7:07pm
8
I uninstalled the debian package, installed the ubuntu one you linked, logged out, logged in, I got the same effect.
stgraber
July 7, 2017, 7:21pm
9
Nope, it’s just me giving you the wrong package… The one I gave you doesn’t have the right cherry-picks…
stgraber
July 7, 2017, 7:21pm
10
euri10
July 7, 2017, 7:41pm
11
➜ ~ lxc-start -n modoboa -o debug -l trace
Success ! thanks for your patience, will report that to debian
stgraber
July 7, 2017, 7:58pm
12
Glad that it worked!
LXCFS 2.0.8 will have the fix for sure, but it’d be great if Debian could cherry-pick it until then.
Stiv
April 19, 2020, 10:08am
13
Hello @stgraber , I have exactly the same problem with cgroups but unfortunately your suggested Ubuntu cherry pick .deb is not available for my armhf hardware.
I am running Debian Stretch on armhf hardware with Armbian OS.
Thanky you
A long time agop I know.