Unprivileged container won't start (cgroups, sysvinit)

sabrina · February 10, 2020, 3:25pm

Hi.
Not sure if I’ll get some echo. As female usually you don’t get any serious feedback. But let’s give it a try:

I’d like to have my unprivileged lxc’s run as lxc-user on my debian buster ( SMP Debian 4.19.67-2+deb10u2) without systemd.

Here is some output of lxc-checkconf
— Namespaces —
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled

— Control groups —
Cgroups: enabled

Cgroup v1 mount points:
/sys/fs/cgroup

Cgroup v2 mount points:

Cgroup v1 systemd controller: missing
Cgroup v1 freezer controller: missing
Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

— Misc —
Veth pair device: enabled, not loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, loaded
CONFIG_NF_NAT_IPV4: enabled, not loaded
CONFIG_NF_NAT_IPV6: enabled, not loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, loaded

— Checkpoint/Restore —
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities:

When i try to start a container i get in trouble with cgroups and apparmor

lxc-start: debian: cgroups/cgfsng.c: cg_hybrid_get_controllers: 746 Found hierarchy not under /sys/fs/cgroup: "/sys/fs/cgroup rw,relatime - cgroup cgroup rw,cpuset,cpu,cpuacct,blkio,memory,devices,freezer,net_cls,perf_event,net_prio,pids,rdma
"
lxc-start: debian: lsm/apparmor.c: make_apparmor_namespace: 761 Permission denied - Error creating AppArmor namespace: /sys/kernel/security/apparmor/policy/namespaces/lxc-debian_<-home-lxc-user-.local-share-lxc>
lxc-start: debian: lsm/apparmor.c: apparmor_prepare: 980 Failed to load generated AppArmor profile
lxc-start: debian: start.c: lxc_init: 899 Failed to initialize LSM
lxc-start: debian: start.c: __lxc_start: 1917 Failed to initialize container “debian”

cat /proc/self/cgroup
1:cpuset,cpu,cpuacct,blkio,memory,devices,freezer,net_cls,perf_event,net_prio,pids,rdma:/

I have already tried out lots of stuff.
Hope someone can point me in the right direction please.

Thank you & Cheers,
Sabrina

stgraber · February 10, 2020, 3:43pm

Unprivileged users can’t create apparmor namespaces, so that appears to be the issue.

So lxc.apparmor.profile = generated won’t work.
lxc.apparmor.profile = lxc-container-default should work as should unconfined or unchanged.

sabrina · February 10, 2020, 4:00pm

Hello!

Thank you!
Well, you got me there. I had that on the other test configs…

The most important problem remains the cgroup error:

lxc-start -n debian -F
lxc-start: debian: cgroups/cgfsng.c: cg_hybrid_get_controllers: 746 Found hierarchy not under /sys/fs/cgroup: "/sys/fs/cgroup rw,relatime - cgroup cgroup rw,cpuset,cpu,cpuacct,blkio,memory,devices,freezer,net_cls,perf_event,net_prio,pids,rdma
" Segmentation fault

Each time i try to “play” with cgroups i get a no space left on device error:

echo 4589 > /sys/fs/cgroup/freezer/0/tasks
-bash: echo: write error: No space left on device

So i am really, really stuck there.
Can you help me please?

stgraber · February 11, 2020, 4:36am

Can you run with -l trace -o debug and paste the content of the debug file?

Also what version of LXC is this?

sabrina · February 11, 2020, 12:23pm

Hi!

here is the output of -l trace -o

lxc-start debian 20200214041404.846 INFO lxc-start debian 20200214041404.846 INFO lxc-start debian 20200214041404.846 INFO lxc-start debian 20200214041404.846 TRACE lxc-start debian 20200214041404.846 TRACE lxc-start debian 20200214041404.846 TRACE lxc-start debian 20200214041404.846 TRACE lxc-start debian 20200214041404.846 TRACE lxc-start debian 20200214041404.847 INFO lxc-start debian 20200214041404.848 TRACE lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 TRACE lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 TRACE lxc-start debian 20200214041404.848 TRACE lxc-start debian 20200214041404.848 TRACE lxc-start debian 20200214041404.848 INFO lxc-start debian 20200214041404.848 TRACE lxc-start debian 20200214041404.848 TRACE lxc-start debian 20200214041404.848 TRACE lxc-start debian 20200214041404.848 TRACE lxc-start debian 20200214041404.848 TRACE lxc-start debian 20200214041404.849 DEBUG lxc-start debian 20200214041404.849 TRACE lxc-start debian 20200214041404.849 DEBUG lxc-start debian 20200214041404.908 TRACE lxc-start debian 20200214041404.908 TRACE lxc-start debian 20200214041404.908 TRACE lxc-start debian 20200214041404.908 TRACE utils - utils.c:get_rundir:275 - XDG_RUNTIME_DIR isn’t set in the environment
confile - confile.c:set_config_idmaps:1605 - Read uid map: type u nsid 0 hostid 165537 range 65536
confile - confile.c:set_config_idmaps:1605 - Read uid map: type g nsid 0 hostid 165537 range 65536
commands - commands.c:lxc_cmd:300 - Connection refused - Command “get_init_pid” failed to connect command socket
commands - commands.c:lxc_cmd:300 - Connection refused - Command “get_state” failed to connect command socket
start - start.c:lxc_init_handler:748 - Created anonymous pair {4,5} of unix sockets
commands - commands.c:lxc_cmd_init:1248 - Creating abstract unix socket “/home/lxc-user/.local/share/lxc/debian/command”
start - start.c:lxc_init_handler:760 - Unix domain socket 6 for command server is ready
lxccontainer - lxccontainer.c:do_lxcapi_start:961 - Set process title to [lxc monitor] /home/lxc-user/.local/share/lxc debian
start - start.c:lxc_start:2064 - Doing lxc_start
lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
start - start.c:lxc_init:777 - Initialized LSM
seccomp - seccomp.c:parse_config_v2:759 - Processing “reject_force_umount # comment this to allow umount -f; not recommended”
seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for reject_force_umount action 0(kill)
seccomp - seccomp.c:parse_config_v2:759 - Processing “[all]”
seccomp - seccomp.c:parse_config_v2:759 - Processing “kexec_load errno 1”
seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for kexec_load action 327681(errno)
seccomp - seccomp.c:parse_config_v2:759 - Processing “open_by_handle_at errno 1”
seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
seccomp - seccomp.c:parse_config_v2:759 - Processing “init_module errno 1”
seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for init_module action 327681(errno)
seccomp - seccomp.c:parse_config_v2:759 - Processing “finit_module errno 1”
seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for finit_module action 327681(errno)
seccomp - seccomp.c:parse_config_v2:759 - Processing “delete_module errno 1”
seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for delete_module action 327681(errno)
seccomp - seccomp.c:parse_config_v2:970 - Merging compat seccomp contexts into main context
start - start.c:lxc_init:784 - Read seccomp policy
start - start.c:lxc_serve_state_clients:466 - Set container state to STARTING
start - start.c:lxc_serve_state_clients:469 - No state clients registered
utils - utils.c:get_rundir:275 - XDG_RUNTIME_DIR isn’t set in the environment
start - start.c:lxc_init:792 - Set container state to “STARTING”
start - start.c:lxc_init:855 - Set environment variables
start - start.c:lxc_init:862 - Ran pre-start hooks
start - start.c:setup_signal_fd:359 - Created signal file descriptor 7
start - start.c:lxc_init:873 - Set up signal fd
terminal - terminal.c:lxc_terminal_peer_default:707 - No such device - The process does not have a controlling terminal
start - start.c:lxc_init:881 - Created console
conf - conf.c:chown_mapped_root:3190 - trying to chown “/dev/pts/1” to 1000
terminal - terminal.c:lxc_terminal_map_ids:1225 - Chowned terminal “/dev/pts/1”
start - start.c:lxc_init:888 - Chowned console
cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1031 - basecginfo is:
cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1032 - 1:cpuset,cpu,cpuacct,blkio,memory,devices,freezer,net_cls,perf_event,net_prio,pids,rdma:/

lxc-start debian 20200214041404.908 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1035 - kernel subsystem 0: cpuset
lxc-start debian 20200214041404.908 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1035 - kernel subsystem 1: cpu
lxc-start debian 20200214041404.908 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1035 - kernel subsystem 2: cpuacct
lxc-start debian 20200214041404.908 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1035 - kernel subsystem 3: blkio
lxc-start debian 20200214041404.908 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1035 - kernel subsystem 4: memory
lxc-start debian 20200214041404.908 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1035 - kernel subsystem 5: devices
lxc-start debian 20200214041404.908 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1035 - kernel subsystem 6: freezer
lxc-start debian 20200214041404.908 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1035 - kernel subsystem 7: net_cls
lxc-start debian 20200214041404.908 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1035 - kernel subsystem 8: perf_event
lxc-start debian 20200214041404.908 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1035 - kernel subsystem 9: net_prio
lxc-start debian 20200214041404.908 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1035 - kernel subsystem 10: pids
lxc-start debian 20200214041404.908 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1035 - kernel subsystem 11: rdma
lxc-start debian 20200214041404.908 ERROR cgfsng - cgroups/cgfsng.c:cg_hybrid_get_controllers:746 - Found hierarchy not under /sys/fs/cgroup: "/sys/fs/cgroup rw,relatime - cgroup cgroup rw,cpuset,cpu,cpuacct,blkio,memory,devices,freezer,net_cls,perf_event,net_prio,pids,rdma
"
lxc-start debian 20200214041404.908 TRACE cgfsng - cgroups/cgfsng.c:cg_hybrid_init:2459 - Writable cgroup hierarchies:
lxc-start debian 20200214041404.908 TRACE cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1008 - No hierarchies found
lxc-start debian 20200214041404.908 TRACE cgroup - cgroups/cgroup.c:cgroup_init:56 - Initialized cgroup driver cgfsng
lxc-start debian 20200214041404.908 TRACE cgroup - cgroups/cgroup.c:cgroup_init:59 - Running with legacy cgroup layout
lxc-start debian 20200214041404.908 TRACE start - start.c:lxc_init:895 - Initialized cgroup driver
lxc-start debian 20200214041404.908 TRACE start - start.c:lxc_init:902 - Initialized LSM
lxc-start debian 20200214041404.908 INFO start - start.c:lxc_init:904 - Container “debian” is initialized
lxc-start debian 20200214041404.908 DEBUG lxccontainer - lxccontainer.c:wait_on_daemonized_start:830 - First child 3676 exited
lxc-start debian 20200214041404.908 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:833 - No such file or directory - Failed to receive the container state
lxc-start debian 20200214041404.908 ERROR lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start debian 20200214041404.908 ERROR lxc_start - tools/lxc_start.c:main:333 - To get more details, run the container in foreground mode
lxc-start debian 20200214041404.908 ERROR lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options

Do i have to put specific values in the lxc.container.conf to make it work…? Which…?

lxc-info --version
3.0.3

Thank you so much!

stgraber · February 11, 2020, 2:18pm

Sounds like you may have an odd cgroup layout on this system.
Can you show:

cat /proc/self/mountinfo
cat /proc/self/cgroup
cat /proc/cgroups

sabrina · February 11, 2020, 2:28pm

cat /proc/self/mountinfo
20 25 0:19 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
21 25 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
22 25 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=63582720k,nr_inodes=993480,mode=755
23 22 0:20 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
24 25 0:21 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=13391808k,mode=755
25 0 254:1 / / rw,relatime - ext4 /dev/mapper/name–vg-root rw,errors=remount-ro
26 24 0:22 / /run/lock rw,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,size=5120k
28 22 0:24 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,size=26783552k
29 25 8:2 / /boot rw,relatime - ext2 /dev/sda2 rw
30 20 0:25 / /sys/fs/cgroup rw,relatime - cgroup cgroup rw,cpuset,cpu,cpuacct,blkio,memory,devices,freezer,net_cls,perf_event,net_prio,pids,rdma
31 20 0:7 / /sys/kernel/security rw,relatime - securityfs securityfs rw
33 25 0:27 / /var/lib/lxcfs rw,nosuid,nodev,relatime - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other

cat /proc/self/cgroup
1:cpuset,cpu,cpuacct,blkio,memory,devices,freezer,net_cls,perf_event,net_prio,pids,rdma:/

cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 1 1 1
cpu 1 1 1
cpuacct 1 1 1
blkio 1 1 1
memory 1 1 1
devices 1 1 1
freezer 1 1 1
net_cls 1 1 1
perf_event 1 1 1
net_prio 1 1 1
pids 1 1 1
rdma 1 1 1

stgraber · February 11, 2020, 10:02pm

Right, this kind of cgroup layout will cause you a bunch of issues I expect.

These days the normal expectation is that unless you’re using cgroup2, /sys/fs/cgroup should be a tmpfs mount with one directory per cgroup controller under it.

This looks like:

stgraber@castiana:~$ ls -lh /sys/fs/cgroup/
total 0
dr-xr-xr-x 88 root root  0 Jan 12 19:59 blkio
lrwxrwxrwx  1 root root 11 Jan 12 19:59 cpu -> cpu,cpuacct
lrwxrwxrwx  1 root root 11 Jan 12 19:59 cpuacct -> cpu,cpuacct
dr-xr-xr-x 88 root root  0 Jan 12 19:59 cpu,cpuacct
dr-xr-xr-x 85 root root  0 Jan 12 19:59 cpuset
dr-xr-xr-x 90 root root  0 Jan 12 19:59 devices
dr-xr-xr-x 90 root root  0 Jan 12 19:59 freezer
dr-xr-xr-x 85 root root  0 Jan 12 19:59 hugetlb
dr-xr-xr-x 89 root root  0 Jan 12 19:59 memory
lrwxrwxrwx  1 root root 16 Jan 12 19:59 net_cls -> net_cls,net_prio
dr-xr-xr-x 85 root root  0 Jan 12 19:59 net_cls,net_prio
lrwxrwxrwx  1 root root 16 Jan 12 19:59 net_prio -> net_cls,net_prio
dr-xr-xr-x 85 root root  0 Jan 12 19:59 perf_event
dr-xr-xr-x 88 root root  0 Jan 12 19:59 pids
dr-xr-xr-x 85 root root  0 Jan 12 19:59 rdma
dr-xr-xr-x 89 root root  0 Jan 12 19:59 systemd
dr-xr-xr-x 88 root root  0 Feb 11 11:36 unified

stgraber@castiana:~$ grep cgroup /proc/self/mountinfo 
35 24 0:30 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:9 - tmpfs tmpfs ro,mode=755
36 35 0:31 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime shared:10 - cgroup2 cgroup2 rw
37 35 0:32 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:11 - cgroup cgroup rw,xattr,name=systemd
41 35 0:36 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:16 - cgroup cgroup rw,hugetlb
42 35 0:37 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,memory
43 35 0:38 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:18 - cgroup cgroup rw,devices
44 35 0:39 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime shared:19 - cgroup cgroup rw,pids
45 35 0:40 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime shared:20 - cgroup cgroup rw,rdma
46 35 0:41 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:21 - cgroup cgroup rw,freezer
47 35 0:42 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:22 - cgroup cgroup rw,perf_event
48 35 0:43 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:23 - cgroup cgroup rw,net_cls,net_prio
49 35 0:44 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:24 - cgroup cgroup rw,cpu,cpuacct
50 35 0:45 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:25 - cgroup cgroup rw,cpuset,clone_children
51 35 0:46 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:26 - cgroup cgroup rw,blkio

Your cgroup setup forces LXC 3.0 into a legacy cgroup manager which may be the source of the crash.
In upcoming 4.0, support for non-standard cgroup layouts such as the one you currently have has been entirely removed.

Abood_Ghazal · February 12, 2020, 1:35pm

I have been using privileged containers for a few days successfully but now I want to migrate to unprivileged ones and having trouble getting them going.

sabrina · February 12, 2020, 8:25pm

Thank you very much for your inquiry!
We will be looking into it and hopefully come up with a solution.
I will report in a few days.

sabrina · February 22, 2020, 9:23pm

Hey!
We solved the issue of the cgroups mount with [0].

Thanks again!

0 https://github.com/tianon/cgroupfs-mount