What’s the start error now?
On stdout:
Error: Failed to run: /opt/bats/bin/lxd forkstart foo /var/lib/lxd/containers /var/log/lxd/foo/lxc.conf:Ok, so let’s investigate that cgroup thing as LXD was also complaining about it in your logs.
What does cat /proc/self/cgroup and ls -lh /sys/fs/cgroup look like in the container?
$ cat /proc/self/cgroup
11:blkio:/lxc.payload.lxlc-bwil02
10:perf_event:/lxc.payload.lxlc-bwil02
9:hugetlb:/lxc.payload.lxlc-bwil02
8:memory:/lxc.payload.lxlc-bwil02
7:cpuacct,cpu:/lxc.payload.lxlc-bwil02
6:cpuset:/lxc.payload.lxlc-bwil02
5:freezer:/lxc.payload.lxlc-bwil02
4:pids:/lxc.payload.lxlc-bwil02
3:devices:/lxc.payload.lxlc-bwil02/user.slice
2:net_prio,net_cls:/lxc.payload.lxlc-bwil02
1:name=systemd:/lxc.payload.lxlc-bwil02/user.slice/user-1515800089.slice/session-1825.scope
$ ls -lh /sys/fs/cgroup/
total 0
drwxr-xr-x 3 root root 60 Feb 12 14:36 blkio
drwxr-xr-x 3 root root 60 Feb 12 14:36 cpu,cpuacct
drwxr-xr-x 3 root root 60 Feb 12 14:36 cpuset
drwxr-xr-x 3 root root 60 Feb 12 14:36 devices
drwxr-xr-x 3 root root 60 Feb 12 14:36 freezer
drwxr-xr-x 3 root root 60 Feb 12 14:36 hugetlb
drwxr-xr-x 3 root root 60 Feb 12 14:36 memory
drwxr-xr-x 3 root root 60 Feb 12 14:36 net_cls,net_prio
drwxr-xr-x 3 root root 60 Feb 12 14:36 perf_event
drwxr-xr-x 3 root root 60 Feb 12 14:36 pids
drwxr-xr-x 3 root root 60 Feb 12 14:36 systemdOk, can you show /var/log/lxd/foo/lxc.conf?
lxc.log.file = /var/log/lxd/foo/lxc.log
lxc.log.level = warn
lxc.console.buffer.size = auto
lxc.console.size = auto
lxc.console.logfile = /var/log/lxd/foo/console.log
lxc.cap.drop = sys_time sys_module sys_rawio mac_admin mac_override
lxc.mount.auto = proc:mixed sys:mixed cgroup:mixed
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir,optional 0 0
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional 0 0
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/config sys/kernel/config none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional 0 0
lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = b *:* m
lxc.cgroup.devices.allow = c *:* m
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.devices.allow = c 1:7 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 10:229 rwm
lxc.cgroup.devices.allow = c 10:200 rwm
lxc.mount.entry = proc dev/.lxc/proc proc create=dir,optional 0 0
lxc.mount.entry = sys dev/.lxc/sys sysfs create=dir,optional 0 0
lxc.arch = linux64
lxc.hook.version = 1
lxc.hook.pre-start = /proc/217/exe callhook /var/lib/lxd 1 start
lxc.hook.stop = /opt/bats/bin/lxd callhook /var/lib/lxd 1 stopns
lxc.hook.post-stop = /opt/bats/bin/lxd callhook /var/lib/lxd 1 stop
lxc.tty.max = 0
lxc.uts.name = foo
lxc.mount.entry = /var/lib/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.seccomp.profile = /var/lib/lxd/security/seccomp/foo
lxc.mount.entry = /var/lib/lxd/shmounts/foo dev/.lxd-mounts none bind,create=dir 0 0
lxc.net.0.name = eth0
lxc.net.0.type = phys
lxc.net.0.flags = up
lxc.net.0.link = veth55e68c88
lxc.rootfs.path = dir:/var/lib/lxd/containers/foo/rootfs
lxc.mount.entry = /var/lib/lxd/devices/foo/disk.libmodules.usr-lib-modules-3.10.0--957.el7.x86_64 usr/lib/modules/3.10.0-957.el7.x86_64 none bind,create=dir 0 0Ok, that’s looking correct so far…
Can you do lxc config set foo raw.lxc=lxc.log.level=trace and then lxc start foo and finally, show the content of /var/log/lxd/foo/lxc.log?
lxc foo 20210212210624.254 TRACE commands - commands.c:lxc_cmd:302 - Connection refused - Command "get_state" failed to connect command socket
lxc foo 20210212210624.254 TRACE start - start.c:lxc_init_handler:694 - Created anonymous pair {3,7} of unix sockets
lxc foo 20210212210624.254 TRACE commands - commands.c:lxc_cmd_init:1770 - Created abstract unix socket "/var/lib/lxd/containers/foo/command"
lxc foo 20210212210624.254 TRACE start - start.c:lxc_init_handler:709 - Unix domain socket 9 for command server is ready
lxc foo 20210212210624.255 INFO lxccontainer - lxccontainer.c:do_lxcapi_start:979 - Set process title to [lxc monitor] /var/lib/lxd/containers foo
lxc foo 20210212210624.255 INFO start - start.c:lxc_check_inherited:286 - Closed inherited fd 4
lxc foo 20210212210624.255 INFO start - start.c:lxc_check_inherited:286 - Closed inherited fd 5
lxc foo 20210212210624.255 INFO start - start.c:lxc_check_inherited:286 - Closed inherited fd 6
lxc foo 20210212210624.255 DEBUG lxccontainer - lxccontainer.c:wait_on_daemonized_start:840 - First child 4897 exited
lxc foo 20210212210624.255 TRACE start - start.c:lxc_start:2114 - Doing lxc_start
lxc foo 20210212210624.255 INFO lsm - lsm/lsm.c:lsm_init:40 - Initialized LSM security driver nop
lxc foo 20210212210624.255 TRACE start - start.c:lxc_init:732 - Initialized LSM
lxc foo 20210212210624.255 TRACE start - start.c:lxc_serve_state_clients:438 - Set container state to STARTING
lxc foo 20210212210624.255 TRACE start - start.c:lxc_serve_state_clients:441 - No state clients registered
lxc foo 20210212210624.255 TRACE start - start.c:lxc_init:738 - Set container state to "STARTING"
lxc foo 20210212210624.255 TRACE start - start.c:lxc_init:794 - Set environment variables
lxc foo 20210212210624.255 INFO conf - conf.c:run_script_argv:333 - Executing script "/proc/217/exe callhook /var/lib/lxd 1 start" for container "foo"
lxc foo 20210212210624.255 TRACE conf - conf.c:run_script_argv:376 - Set environment variable: LXC_HOOK_TYPE=pre-start
lxc foo 20210212210624.255 TRACE conf - conf.c:run_script_argv:381 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc foo 20210212210624.313 TRACE start - start.c:lxc_init:799 - Ran pre-start hooks
lxc foo 20210212210624.313 TRACE start - start.c:setup_signal_fd:331 - Created signal file descriptor 5
lxc foo 20210212210624.313 TRACE start - start.c:lxc_init:808 - Set up signal fd
lxc foo 20210212210624.314 DEBUG terminal - terminal.c:lxc_terminal_peer_default:665 - No such device - The process does not have a controlling terminal
lxc foo 20210212210624.314 DEBUG terminal - terminal.c:lxc_terminal_create_log_file:837 - Using "/var/log/lxd/foo/console.log" as terminal log file
lxc foo 20210212210624.314 TRACE terminal - terminal.c:lxc_terminal_create_ringbuf:818 - Allocated 131072 byte terminal ringbuffer
lxc foo 20210212210624.314 TRACE start - start.c:lxc_init:816 - Created console
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:918 - basecginfo is:
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:919 - 11:blkio:/lxc.payload.lxlc-bwil02
10:perf_event:/lxc.payload.lxlc-bwil02
9:hugetlb:/lxc.payload.lxlc-bwil02
8:memory:/lxc.payload.lxlc-bwil02
7:cpuacct,cpu:/lxc.payload.lxlc-bwil02
6:cpuset:/lxc.payload.lxlc-bwil02
5:freezer:/lxc.payload.lxlc-bwil02
4:pids:/lxc.payload.lxlc-bwil02
3:devices:/lxc.payload.lxlc-bwil02
2:net_prio,net_cls:/lxc.payload.lxlc-bwil02
1:name=systemd:/lxc.payload.lxlc-bwil02
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:922 - kernel subsystem 0: blkio
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:922 - kernel subsystem 1: perf_event
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:922 - kernel subsystem 2: hugetlb
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:922 - kernel subsystem 3: memory
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:922 - kernel subsystem 4: cpuacct
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:922 - kernel subsystem 5: cpu
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:922 - kernel subsystem 6: cpuset
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:922 - kernel subsystem 7: freezer
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:922 - kernel subsystem 8: pids
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:922 - kernel subsystem 9: devices
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:922 - kernel subsystem 10: net_prio
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:922 - kernel subsystem 11: net_cls
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:925 - named subsystem 0: name=systemd
lxc foo 20210212210624.314 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3261 - The /lxc.payload.lxlc-bwil02 group is not writeable
lxc foo 20210212210624.314 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212210624.314 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212210624.314 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212210624.314 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3261 - The /lxc.payload.lxlc-bwil02 group is not writeable
lxc foo 20210212210624.314 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212210624.314 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212210624.314 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212210624.314 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3297 - Writable cgroup hierarchies:
lxc foo 20210212210624.314 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:895 - No hierarchies found
lxc foo 20210212210624.314 TRACE cgroup - cgroups/cgroup.c:cgroup_init:41 - Initialized cgroup driver cgfsng
lxc foo 20210212210624.314 TRACE cgroup - cgroups/cgroup.c:cgroup_init:44 - Running with legacy cgroup layout
lxc foo 20210212210624.314 TRACE start - start.c:lxc_init:823 - Initialized cgroup driver
lxc foo 20210212210624.314 TRACE start - start.c:lxc_init:828 - Read seccomp policy
lxc foo 20210212210624.314 TRACE start - start.c:lxc_init:835 - Initialized LSM
lxc foo 20210212210624.314 INFO start - start.c:lxc_init:837 - Container "foo" is initialized
lxc foo 20210212210624.314 TRACE sync - sync.c:lxc_sync_init:113 - Initialized synchronization infrastructure
lxc foo 20210212210624.314 TRACE start - start.c:lxc_spawn:1652 - Function not implemented - Failed to spawn container directly into target cgroup
lxc foo 20210212210624.314 TRACE start - start.c:lxc_spawn:1668 - Function not implemented - Failed to spawn container via clone3()
lxc foo 20210212210624.318 TRACE start - start.c:lxc_spawn:1684 - Cloned child process 4905
lxc foo 20210212210624.318 ERROR utils - utils.c:lxc_can_use_pidfd:1905 - Kernel does not support pidfds
lxc foo 20210212210624.318 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWNS
lxc foo 20210212210624.318 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWPID
lxc foo 20210212210624.318 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWUTS
lxc foo 20210212210624.318 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWIPC
lxc foo 20210212210624.318 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWNET
lxc foo 20210212210624.318 DEBUG start - start.c:lxc_try_preserve_namespaces:167 - Preserved mnt namespace via fd 17
lxc foo 20210212210624.318 DEBUG start - start.c:lxc_try_preserve_namespaces:167 - Preserved pid namespace via fd 18
lxc foo 20210212210624.318 DEBUG start - start.c:lxc_try_preserve_namespaces:167 - Preserved uts namespace via fd 19
lxc foo 20210212210624.318 DEBUG start - start.c:lxc_try_preserve_namespaces:167 - Preserved ipc namespace via fd 20
lxc foo 20210212210624.318 DEBUG start - start.c:lxc_try_preserve_namespaces:167 - Preserved net namespace via fd 21
lxc foo 20210212210624.318 TRACE sync - sync.c:lxc_sync_wake_child:96 - Child waking parent with sequence startup
lxc foo 20210212210624.318 TRACE sync - sync.c:lxc_sync_wait_child:90 - Child waiting for parent with sequence configure
lxc foo 20210212210624.318 TRACE sync - sync.c:lxc_sync_wait_parent:84 - Parent waiting for child with sequence startup
lxc foo 20210212210624.318 TRACE sync - sync.c:lxc_sync_barrier_parent:65 - Child waking parent with sequence configure and waiting for sequence post-configure
lxc foo 20210212210624.318 ERROR start - start.c:lxc_spawn:1741 - Failed to setup cgroup limits for container "foo"
lxc foo 20210212210624.318 DEBUG network - network.c:lxc_delete_network:3672 - Deleted network devices
lxc foo 20210212210624.318 TRACE start - start.c:lxc_serve_state_socket_pair:503 - Sent container state "ABORTING" to 7
lxc foo 20210212210624.318 TRACE start - start.c:lxc_serve_state_clients:438 - Set container state to ABORTING
lxc foo 20210212210624.318 TRACE start - start.c:lxc_serve_state_clients:441 - No state clients registered
lxc foo 20210212210624.318 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:860 - Received container state "ABORTING" instead of "RUNNING"
lxc foo 20210212210624.318 ERROR start - start.c:__lxc_start:1999 - Failed to spawn container "foo"
lxc foo 20210212210624.318 TRACE start - start.c:lxc_serve_state_clients:438 - Set container state to ABORTING
lxc foo 20210212210624.318 TRACE start - start.c:lxc_serve_state_clients:441 - No state clients registered
lxc foo 20210212210624.318 WARN start - start.c:lxc_abort:1018 - No such process - Failed to send SIGKILL to 4905
lxc foo 20210212210624.318 TRACE start - start.c:lxc_serve_state_clients:438 - Set container state to STOPPING
lxc foo 20210212210624.318 TRACE start - start.c:lxc_serve_state_clients:441 - No state clients registered
lxc foo 20210212210624.318 TRACE start - start.c:lxc_end:893 - Set environment variable LXC_MNT_NS=/proc/4898/fd/17
lxc foo 20210212210624.318 TRACE start - start.c:lxc_end:893 - Set environment variable LXC_PID_NS=/proc/4898/fd/18
lxc foo 20210212210624.318 TRACE start - start.c:lxc_end:893 - Set environment variable LXC_UTS_NS=/proc/4898/fd/19
lxc foo 20210212210624.318 TRACE start - start.c:lxc_end:893 - Set environment variable LXC_IPC_NS=/proc/4898/fd/20
lxc foo 20210212210624.318 TRACE start - start.c:lxc_end:893 - Set environment variable LXC_NET_NS=/proc/4898/fd/21
lxc foo 20210212210624.318 INFO conf - conf.c:run_script_argv:333 - Executing script "/opt/bats/bin/lxd callhook /var/lib/lxd 1 stopns" for container "foo"
lxc foo 20210212210624.318 TRACE conf - conf.c:run_script_argv:376 - Set environment variable: LXC_HOOK_TYPE=stop
lxc foo 20210212210624.318 TRACE conf - conf.c:run_script_argv:381 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc foo 20210212210624.447 TRACE start - start.c:lxc_end:936 - Closed command socket
lxc foo 20210212210624.447 TRACE start - start.c:lxc_end:947 - Set container state to "STOPPED"
lxc foo 20210212210624.466 INFO conf - conf.c:run_script_argv:333 - Executing script "/opt/bats/bin/lxd callhook /var/lib/lxd 1 stop" for container "foo"
lxc foo 20210212210624.466 TRACE conf - conf.c:run_script_argv:376 - Set environment variable: LXC_HOOK_TYPE=post-stop
lxc foo 20210212210624.466 TRACE conf - conf.c:run_script_argv:381 - Set environment variable: LXC_HOOK_SECTION=lxcOh, hold on, there’s definitely something odd there…
Your kernel doesn’t support cgroup namespacing. That’s not a problem for the first level container, we’re faking a normal hierarchy well enough, but it looks like our current cgroup handling logic doesn’t like it when trying to nest on such a kernel…
@brauner looks like we may have regressed this with the recent cgroup rework?
I realize our 3.10.x RHEL kernel is a bit old these days, but we’ll likely be running it for another year on most systems.
Is there more information I can provide to resolve this and allow us to upgrade LXD?
Thanks!
I’ve mentioned it to @brauner in a meeting today and he’s doing a whole bunch of cgroup fixes in liblxc, maybe some of that will help, if not, he knows about this particular case and will investigate it soon.
If you have some spare time, you can try building LXC from the master branch and test that to see if we haven’t already fixed it somehow 
Ah, sorry I replied by mail but somehow discuss still doesn’t handle this correctly. Anyway, I’m fixing this is part of a chunk of larger rework.
Ok, current master after my slight rework:
root 2742 0.0 0.2 1198724 14000 ? Ss 14:13 0:00 [lxc monitor] /var/lib/lxd/containers h1
1000000 2750 0.3 0.2 171584 12728 ? Ss 14:13 0:00 \_ /sbin/init
1000000 2810 0.1 0.2 39272 13080 ? Ss 14:13 0:00 \_ /usr/lib/systemd/systemd-journald
1000000 2833 0.0 0.1 26332 6788 ? Ss 14:13 0:00 \_ /usr/lib/systemd/systemd-udevd
1000100 2864 0.1 0.2 28676 9556 ? Ss 14:13 0:00 \_ /usr/lib/systemd/systemd-networkd
1000000 2868 0.0 0.0 11296 2872 ? Ss 14:13 0:00 \_ /usr/sbin/cron -f
1000103 2869 0.0 0.0 8648 4264 ? Ss 14:13 0:00 \_ /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
1000000 2873 0.0 0.4 34232 19236 ? Ss 14:13 0:00 \_ /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
1000104 2875 0.0 0.1 78896 4912 ? Ssl 14:13 0:00 \_ /usr/sbin/rsyslogd -n -iNONE
1000000 2877 0.1 0.1 20060 8636 ? Ss 14:13 0:00 \_ /usr/lib/systemd/systemd-logind
1000101 2914 0.1 0.2 25340 12780 ? Ss 14:13 0:00 \_ /usr/lib/systemd/systemd-resolved
1000000 2927 0.0 0.0 10084 2400 pts/3 Ss+ 14:13 0:00 \_ /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 vt220
1000000 2971 0.0 0.2 21164 10320 ? Ss 14:14 0:00 \_ /usr/lib/systemd/systemd --user
1000000 2972 0.0 0.0 107548 2900 ? S 14:14 0:00 | \_ (sd-pam)
1000000 3045 0.0 0.0 6088 3412 ? Ss 14:15 0:00 \_ [lxc monitor] /var/lib/lxc h1
1100000 3046 0.4 0.2 100056 10040 ? Ss 14:15 0:00 \_ /sbin/init
1100000 3097 0.2 0.2 32144 10660 ? Ss 14:15 0:00 \_ /lib/systemd/systemd-journald
1100000 3136 0.0 0.1 20748 4812 ? Ss 14:15 0:00 \_ /lib/systemd/systemd-udevd
1100100 3141 0.2 0.1 14644 6096 ? Ss 14:15 0:00 \_ /lib/systemd/systemd-networkd
1100000 3148 0.0 0.0 9432 2812 ? Ss 14:15 0:00 \_ /usr/sbin/cron -f
1100103 3149 0.0 0.0 8612 4428 ? Ss 14:15 0:00 \_ /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
1100000 3152 0.3 0.4 33640 19632 ? Ss 14:15 0:00 \_ /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
1100104 3153 0.0 0.1 152628 4932 ? Ssl 14:15 0:00 \_ /usr/sbin/rsyslogd -n -iNONE
1100000 3154 0.3 0.1 13540 6016 ? Ss 14:15 0:00 \_ /lib/systemd/systemd-logind
1100101 3157 0.2 0.2 24192 12364 ? Ss 14:15 0:00 \_ /lib/systemd/systemd-resolved
1100000 3165 0.0 0.0 8192 2248 pts/1 Ss+ 14:15 0:00 \_ /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 vt220
1100000 3166 0.0 0.0 8192 2068 pts/0 Ss+ 14:15 0:00 \_ /sbin/agetty -o -p -- \u --noclear --keep-baud pts/0 115200,38400,9600 vt220
1100000 3167 0.0 0.0 8192 2312 pts/1 Ss+ 14:15 0:00 \_ /sbin/agetty -o -p -- \u --noclear --keep-baud pts/1 115200,38400,9600 vt220
1100000 3168 0.0 0.0 8192 2084 pts/2 Ss+ 14:15 0:00 \_ /sbin/agetty -o -p -- \u --noclear --keep-baud pts/2 115200,38400,9600 vt220
1100000 3169 0.0 0.0 8192 2232 pts/3 Ss+ 14:15 0:00 \_ /sbin/agetty -o -p -- \u --noclear --keep-baud pts/3 115200,38400,9600 vt220
brauner@f2-vm|~
> cat /proc/2742/cgroup
12:rdma:/lxc.monitor.h1
11:blkio:/lxc.monitor.h1
10:net_cls,net_prio:/lxc.monitor.h1
9:cpuset:/lxc.monitor.h1
8:freezer:/lxc.monitor.h1
7:hugetlb:/lxc.monitor.h1
6:perf_event:/lxc.monitor.h1
5:memory:/lxc.monitor.h1
4:devices:/lxc.monitor.h1
3:cpu,cpuacct:/lxc.monitor.h1
2:pids:/lxc.monitor.h1
1:name=systemd:/lxc.monitor.h1
0::/
brauner@f2-vm|~
> cat /proc/2750/cgroup
12:rdma:/lxc.payload.h1
11:blkio:/lxc.payload.h1
10:net_cls,net_prio:/lxc.payload.h1
9:cpuset:/lxc.payload.h1
8:freezer:/lxc.payload.h1
7:hugetlb:/lxc.payload.h1
6:perf_event:/lxc.payload.h1
5:memory:/lxc.payload.h1
4:devices:/lxc.payload.h1
3:cpu,cpuacct:/lxc.payload.h1
2:pids:/lxc.payload.h1
1:name=systemd:/lxc.payload.h1/init.scope
0::/
brauner@f2-vm|~
> cat /proc/3045/cgroup
12:rdma:/lxc.payload.h1/lxc.monitor.h1
11:blkio:/lxc.payload.h1/lxc.monitor.h1
10:net_cls,net_prio:/lxc.payload.h1/lxc.monitor.h1
9:cpuset:/lxc.payload.h1/lxc.monitor.h1
8:freezer:/lxc.payload.h1/lxc.monitor.h1
7:hugetlb:/lxc.payload.h1/lxc.monitor.h1
6:perf_event:/lxc.payload.h1/lxc.monitor.h1
5:memory:/lxc.payload.h1/lxc.monitor.h1
4:devices:/lxc.payload.h1/lxc.monitor.h1
3:cpu,cpuacct:/lxc.payload.h1/lxc.monitor.h1
2:pids:/lxc.payload.h1/lxc.monitor.h1
1:name=systemd:/lxc.payload.h1/lxc.monitor.h1
0::/
brauner@f2-vm|~
> cat /proc/3046/cgroup
12:rdma:/lxc.payload.h1/lxc.payload.h1-1
11:blkio:/lxc.payload.h1/lxc.payload.h1-1
10:net_cls,net_prio:/lxc.payload.h1/lxc.payload.h1-1
9:cpuset:/lxc.payload.h1/lxc.payload.h1-1
8:freezer:/lxc.payload.h1/lxc.payload.h1-1
7:hugetlb:/lxc.payload.h1/lxc.payload.h1-1
6:perf_event:/lxc.payload.h1/lxc.payload.h1-1
5:memory:/lxc.payload.h1/lxc.payload.h1-1
4:devices:/lxc.payload.h1/lxc.payload.h1-1
3:cpu,cpuacct:/lxc.payload.h1/lxc.payload.h1-1
2:pids:/lxc.payload.h1/lxc.payload.h1-1
1:name=systemd:/lxc.payload.h1/lxc.payload.h1-1/init.scope
0::/
It’s a pure legacy layout too:
brauner@f2-vm|~
> findmnt | grep -i cgroup
│ ├─/sys/fs/cgroup tmpfs tmpfs ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024,mode=755
│ │ ├─/sys/fs/cgroup/systemd cgroup cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd
│ │ ├─/sys/fs/cgroup/pids cgroup cgroup rw,nosuid,nodev,noexec,relatime,pids
│ │ ├─/sys/fs/cgroup/cpu,cpuacct cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct
│ │ ├─/sys/fs/cgroup/devices cgroup cgroup rw,nosuid,nodev,noexec,relatime,devices
│ │ ├─/sys/fs/cgroup/memory cgroup cgroup rw,nosuid,nodev,noexec,relatime,memory
│ │ ├─/sys/fs/cgroup/perf_event cgroup cgroup rw,nosuid,nodev,noexec,relatime,perf_event
│ │ ├─/sys/fs/cgroup/hugetlb cgroup cgroup rw,nosuid,nodev,noexec,relatime,hugetlb
│ │ ├─/sys/fs/cgroup/freezer cgroup cgroup rw,nosuid,nodev,noexec,relatime,freezer
│ │ ├─/sys/fs/cgroup/cpuset cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpuset
│ │ ├─/sys/fs/cgroup/net_cls,net_prio cgroup cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio
│ │ ├─/sys/fs/cgroup/blkio cgroup cgroup rw,nosuid,nodev,noexec,relatime,blkio
│ │ └─/sys/fs/cgroup/rdma cgroup cgroup rw,nosuid,nodev,noexec,relatime,rdma
I’ll go look into tweaking for old kernels that don’t have cgroup namespaces a bit more.
I rebuilt off master, last commit: cca7d405fe44c755533891dd27d3d273becd419b
$ lxc info --show-log foo
Name: foo
Location: none
Remote: unix://
Architecture: x86_64
Created: 2021/02/12 17:03 UTC
Status: Stopped
Type: container
Profiles: default
Log:
lxc foo 20210218173340.802 TRACE commands - commands.c:lxc_cmd:302 - Connection refused - Command "get_state" failed to connect command socket
lxc foo 20210218173340.802 TRACE start - start.c:lxc_init_handler:712 - Created anonymous pair {3,7} of unix sockets
lxc foo 20210218173340.802 TRACE commands - commands.c:lxc_cmd_init:1793 - Created abstract unix socket "/var/lib/lxd/containers/foo/command"
lxc foo 20210218173340.802 TRACE start - start.c:lxc_init_handler:727 - Unix domain socket 9 for command server is ready
lxc foo 20210218173340.803 INFO lxccontainer - lxccontainer.c:do_lxcapi_start:988 - Set process title to [lxc monitor] /var/lib/lxd/containers foo
lxc foo 20210218173340.803 INFO start - start.c:lxc_check_inherited:296 - Closed inherited fd 4
lxc foo 20210218173340.803 INFO start - start.c:lxc_check_inherited:296 - Closed inherited fd 5
lxc foo 20210218173340.803 INFO start - start.c:lxc_check_inherited:296 - Closed inherited fd 6
lxc foo 20210218173340.803 DEBUG lxccontainer - lxccontainer.c:wait_on_daemonized_start:849 - First child 2505 exited
lxc foo 20210218173340.803 TRACE start - start.c:lxc_start:2177 - Doing lxc_start
lxc foo 20210218173340.803 INFO lsm - lsm/lsm.c:lsm_init_static:40 - Initialized LSM security driver nop
lxc foo 20210218173340.803 TRACE start - start.c:lxc_init:750 - Initialized LSM
lxc foo 20210218173340.803 TRACE start - start.c:lxc_serve_state_clients:448 - Set container state to STARTING
lxc foo 20210218173340.803 TRACE start - start.c:lxc_serve_state_clients:451 - No state clients registered
lxc foo 20210218173340.804 TRACE start - start.c:lxc_init:756 - Set container state to "STARTING"
lxc foo 20210218173340.804 TRACE start - start.c:lxc_init:812 - Set environment variables
lxc foo 20210218173340.804 INFO conf - conf.c:run_script_argv:333 - Executing script "/proc/228/exe callhook /var/lib/lxd 1 start" for container "foo"
lxc foo 20210218173340.804 TRACE conf - conf.c:run_script_argv:376 - Set environment variable: LXC_HOOK_TYPE=pre-start
lxc foo 20210218173340.804 TRACE conf - conf.c:run_script_argv:381 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc foo 20210218173340.876 TRACE start - start.c:lxc_init:817 - Ran pre-start hooks
lxc foo 20210218173340.876 TRACE start - start.c:setup_signal_fd:341 - Created signal file descriptor 5
lxc foo 20210218173340.876 TRACE start - start.c:lxc_init:826 - Set up signal fd
lxc foo 20210218173340.876 DEBUG terminal - terminal.c:lxc_terminal_peer_default:665 - No such device - The process does not have a controlling terminal
lxc foo 20210218173340.876 DEBUG terminal - terminal.c:lxc_terminal_create_log_file:837 - Using "/var/log/lxd/foo/console.log" as terminal log file
lxc foo 20210218173340.876 TRACE terminal - terminal.c:lxc_terminal_create_ringbuf:818 - Allocated 131072 byte terminal ringbuffer
lxc foo 20210218173340.876 TRACE start - start.c:lxc_init:834 - Created console
lxc foo 20210218173340.876 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:775 - basecginfo is:
lxc foo 20210218173340.876 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:776 - 11:blkio:/lxc.payload.lxlc-bwil02
10:perf_event:/lxc.payload.lxlc-bwil02
9:hugetlb:/lxc.payload.lxlc-bwil02
8:memory:/lxc.payload.lxlc-bwil02
7:cpuacct,cpu:/lxc.payload.lxlc-bwil02
6:cpuset:/lxc.payload.lxlc-bwil02
5:freezer:/lxc.payload.lxlc-bwil02
4:pids:/lxc.payload.lxlc-bwil02
3:devices:/lxc.payload.lxlc-bwil02
2:net_prio,net_cls:/lxc.payload.lxlc-bwil02
1:name=systemd:/lxc.payload.lxlc-bwil02
lxc foo 20210218173340.876 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:779 - kernel subsystem 0: blkio
lxc foo 20210218173340.876 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:779 - kernel subsystem 1: perf_event
lxc foo 20210218173340.876 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:779 - kernel subsystem 2: hugetlb
lxc foo 20210218173340.876 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:779 - kernel subsystem 3: memory
lxc foo 20210218173340.876 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:779 - kernel subsystem 4: cpuacct
lxc foo 20210218173340.876 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:779 - kernel subsystem 5: cpu
lxc foo 20210218173340.876 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:779 - kernel subsystem 6: cpuset
lxc foo 20210218173340.876 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:779 - kernel subsystem 7: freezer
lxc foo 20210218173340.876 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:779 - kernel subsystem 8: pids
lxc foo 20210218173340.876 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:779 - kernel subsystem 9: devices
lxc foo 20210218173340.876 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:779 - kernel subsystem 10: net_prio
lxc foo 20210218173340.876 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:779 - kernel subsystem 11: net_cls
lxc foo 20210218173340.876 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:782 - named subsystem 0: name=systemd
lxc foo 20210218173340.877 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3457 - Failed to find current cgroup
lxc foo 20210218173340.877 TRACE cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3466 - The lxc.payload.lxlc-bwil02 group is not writeable
lxc foo 20210218173340.877 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3457 - Failed to find current cgroup
lxc foo 20210218173340.877 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3457 - Failed to find current cgroup
lxc foo 20210218173340.877 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3457 - Failed to find current cgroup
lxc foo 20210218173340.877 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3457 - Failed to find current cgroup
lxc foo 20210218173340.877 TRACE cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3466 - The lxc.payload.lxlc-bwil02 group is not writeable
lxc foo 20210218173340.877 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3457 - Failed to find current cgroup
lxc foo 20210218173340.877 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3457 - Failed to find current cgroup
lxc foo 20210218173340.877 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3457 - Failed to find current cgroup
lxc foo 20210218173340.877 WARN cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3457 - Failed to find current cgroup
lxc foo 20210218173340.877 ERROR cgroup - cgroups/cgroup.c:cgroup_init:38 - No such file or directory - No cgroup hierarchies found
lxc foo 20210218173340.877 ERROR start - start.c:lxc_init:838 - Failed to initialize cgroup driver
lxc foo 20210218173340.877 ERROR start - start.c:__lxc_start:2010 - Failed to initialize container "foo"
lxc foo 20210218173340.877 TRACE start - start.c:lxc_serve_state_clients:448 - Set container state to ABORTING
lxc foo 20210218173340.877 TRACE start - start.c:lxc_serve_state_clients:451 - No state clients registered
lxc foo 20210218173340.877 TRACE start - start.c:lxc_serve_state_clients:448 - Set container state to STOPPING
lxc foo 20210218173340.877 TRACE start - start.c:lxc_serve_state_clients:451 - No state clients registered
lxc foo 20210218173340.877 INFO conf - conf.c:run_script_argv:333 - Executing script "/opt/bats/bin/lxd callhook /var/lib/lxd 1 stopns" for container "foo"
lxc foo 20210218173340.877 TRACE conf - conf.c:run_script_argv:376 - Set environment variable: LXC_HOOK_TYPE=stop
lxc foo 20210218173340.877 TRACE conf - conf.c:run_script_argv:381 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc foo 20210218173410.915 DEBUG conf - conf.c:run_buffer:303 - Script exec /opt/bats/bin/lxd callhook /var/lib/lxd 1 stopns produced output: Error: Hook didn't finish within 30s
lxc foo 20210218173410.916 ERROR conf - conf.c:run_buffer:314 - Script exited with status 1
lxc foo 20210218173410.916 ERROR start - start.c:lxc_end:921 - Failed to run "lxc.hook.stop" hook
lxc foo 20210218173410.916 TRACE start - start.c:lxc_end:940 - Closed command socket
lxc foo 20210218173410.916 TRACE start - start.c:lxc_end:951 - Set container state to "STOPPED"
lxc foo 20210218173410.916 INFO conf - conf.c:run_script_argv:333 - Executing script "/opt/bats/bin/lxd callhook /var/lib/lxd 1 stop" for container "foo"
lxc foo 20210218173410.916 TRACE conf - conf.c:run_script_argv:376 - Set environment variable: LXC_HOOK_TYPE=post-stop
lxc foo 20210218173410.916 TRACE conf - conf.c:run_script_argv:381 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc foo 20210218173410.972 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:860 - No such file or directory - Failed to receive the container state
lxc 20210218173410.972 WARN commands - commands.c:lxc_cmd_rsp_recv:126 - Connection reset by peer - Failed to receive response for command "get_state"
lxc 20210218173410.972 WARN commands - commands.c:lxc_cmd_rsp_recv:126 - Connection reset by peer - Failed to receive response for command "get_state"
lxc 20210218173410.972 WARN commands - commands.c:lxc_cmd_rsp_recv:126 - Connection reset by peer - Failed to receive response for command "get_init_pid"
lxc foo 20210218173340.877 ERROR cgroup - cgroups/cgroup.c:cgroup_init:38 - No such file or directory - No cgroup hierarchies found
Is that with ^^ this version too @stgraber?
Is current master with f640c8187aac72d303d2edbdc25d7b1082e330e7
affected?
Because I kept running a tightloop on both unified and hybrid and things were fine. Let me try nesting again.
This is a non-nested container?