On a clean Alpine (edge) with LXC v4.0.12-r2 installed (following instructions at LXC - Alpine Linux ), as an unprivileged user, I’ve created the following file:
localhost:~$ cat ~/.config/lxc/simple-alpine-nested.conf
lxc.idmap = u 0 11500000 500000
lxc.idmap = g 0 11500000 500000
lxc.include = /usr/share/lxc/config/nesting.conf
lxc.apparmor.allow_nesting = 1
lxc.seccomp.allow_nesting = 1
lxc.mount.auto = cgroup:rw:force
For reference:
localhost:~$ cat /usr/share/lxc/config/nesting.conf
# Use a profile which allows nesting
lxc.apparmor.profile = lxc-container-default-with-nesting
# Add uncovered mounts of proc and sys, else unprivileged users
# cannot remount those
lxc.mount.entry = proc dev/.lxc/proc proc create=dir,optional 0 0
lxc.mount.entry = sys dev/.lxc/sys sysfs create=dir,optional 0 0
localhost:~$ cat /etc/apparmor.d/lxc/lxc-default-with-nesting
# Do not load this file. Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc
profile lxc-container-default-with-nesting flags=(attach_disconnected,mediate_deleted) {
#include <abstractions/lxc/container-base>
#include <abstractions/lxc/start-container>
deny /dev/.lxc/proc/** rw,
deny /dev/.lxc/sys/** rw,
mount fstype=proc -> /var/cache/lxc/**,
mount fstype=sysfs -> /var/cache/lxc/**,
mount options=(rw,bind),
mount fstype=cgroup -> /sys/fs/cgroup/**,
mount fstype=cgroup2 -> /sys/fs/cgroup/**,
}
Then I’ve executed lxc-create -n simple-alpine-nested -f ~/.config/lxc/simple-alpine-nested.conf -t download, and picked alpine/3.16/amd64.
For reference, generated config of the created container:
localhost:~$ cat ~/.local/share/lxc/simple-alpine-nested/config
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)
# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)
# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.include = /usr/share/lxc/config/userns.conf
lxc.arch = linux64
# Container specific configuration
lxc.idmap = u 0 11500000 500000
lxc.idmap = g 0 11500000 500000
lxc.include = /usr/share/lxc/config/nesting.conf
lxc.apparmor.allow_nesting = 1
lxc.seccomp.allow_nesting = 1
lxc.mount.auto = cgroup:rw:force
lxc.rootfs.path = dir:/home/inga/.local/share/lxc/simple-alpine-nested/rootfs
lxc.uts.name = simple-alpine-nested
# Network configuration
localhost:~$
Then:
localhost:~$ lxc-start -n simple-alpine-nested
localhost:~$ lxc-attach --clear-env -n simple-alpine-nested
/ # ls -la /sys/fs/cgroup/
total 0
dr-xr-xr-x 2 nobody nobody 0 Sep 4 03:15 .
drwxr-xr-x 7 nobody nobody 0 Sep 4 03:15 ..
/ #
However, according to all documentation and tutorials I was able to find on the internet (although none of them covered alpine specifically), /sys/fs/cgroup should be populated in the container.
And I’ve tried everything I could think of over the last two days to make it work, to no effect.
The reason for trying to run this configuration is to make docker work inside a container; without /sys/fs/cgroups it produces a bunch of errors and does not start.
Logs from container start attempt; unfortunately, I don’t know enough to understand what’s going wrong here (and don’t know how else to debug this issue):
localhost:~$ lxc-start -n simple-alpine-nested --logfile lxcstartnested.log --logpriority info
localhost:~$ cat lxcstartnested.log
lxc-start simple-alpine-nested 20220904033500.388 INFO confile - confile.c:set_config_idmaps:2069 - Read uid map: type u nsid 0 hostid 11500000 range 500000
lxc-start simple-alpine-nested 20220904033500.389 INFO confile - confile.c:set_config_idmaps:2069 - Read uid map: type g nsid 0 hostid 11500000 range 500000
lxc-start simple-alpine-nested 20220904033500.389 INFO lxccontainer - lxccontainer.c:do_lxcapi_start:997 - Set process title to [lxc monitor] /home/inga/.local/share/lxc simple-alpine-nested
lxc-start simple-alpine-nested 20220904033500.389 INFO start - start.c:lxc_check_inherited:328 - Closed inherited fd 5
lxc-start simple-alpine-nested 20220904033500.389 INFO start - start.c:lxc_check_inherited:328 - Closed inherited fd 6
lxc-start simple-alpine-nested 20220904033500.390 INFO lsm - lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver nop
lxc-start simple-alpine-nested 20220904033500.390 INFO cgfsng - cgroups/cgfsng.c:legacy_hierarchy_delegated:3060 - Permission denied - Legacy hierarchy not writable, skipping
lxc-start simple-alpine-nested 20220904033500.390 INFO cgfsng - cgroups/cgfsng.c:legacy_hierarchy_delegated:3060 - Permission denied - Legacy hierarchy not writable, skipping
lxc-start simple-alpine-nested 20220904033500.390 INFO cgfsng - cgroups/cgfsng.c:legacy_hierarchy_delegated:3060 - Permission denied - Legacy hierarchy not writable, skipping
lxc-start simple-alpine-nested 20220904033500.390 INFO cgfsng - cgroups/cgfsng.c:legacy_hierarchy_delegated:3060 - Permission denied - Legacy hierarchy not writable, skipping
lxc-start simple-alpine-nested 20220904033500.390 INFO cgfsng - cgroups/cgfsng.c:legacy_hierarchy_delegated:3060 - Permission denied - Legacy hierarchy not writable, skipping
lxc-start simple-alpine-nested 20220904033500.390 INFO cgfsng - cgroups/cgfsng.c:legacy_hierarchy_delegated:3060 - Permission denied - Legacy hierarchy not writable, skipping
lxc-start simple-alpine-nested 20220904033500.390 INFO cgfsng - cgroups/cgfsng.c:legacy_hierarchy_delegated:3060 - Permission denied - Legacy hierarchy not writable, skipping
lxc-start simple-alpine-nested 20220904033500.390 INFO cgfsng - cgroups/cgfsng.c:legacy_hierarchy_delegated:3060 - Permission denied - Legacy hierarchy not writable, skipping
lxc-start simple-alpine-nested 20220904033500.390 INFO cgfsng - cgroups/cgfsng.c:legacy_hierarchy_delegated:3060 - Permission denied - Legacy hierarchy not writable, skipping
lxc-start simple-alpine-nested 20220904033500.390 INFO cgfsng - cgroups/cgfsng.c:legacy_hierarchy_delegated:3060 - Permission denied - Legacy hierarchy not writable, skipping
lxc-start simple-alpine-nested 20220904033500.390 INFO cgfsng - cgroups/cgfsng.c:legacy_hierarchy_delegated:3060 - Permission denied - Legacy hierarchy not writable, skipping
lxc-start simple-alpine-nested 20220904033500.390 INFO cgfsng - cgroups/cgfsng.c:legacy_hierarchy_delegated:3060 - Permission denied - Legacy hierarchy not writable, skipping
lxc-start simple-alpine-nested 20220904033500.390 INFO cgfsng - cgroups/cgfsng.c:legacy_hierarchy_delegated:3060 - Permission denied - Legacy hierarchy not writable, skipping
lxc-start simple-alpine-nested 20220904033500.390 INFO cgfsng - cgroups/cgfsng.c:legacy_hierarchy_delegated:3060 - Permission denied - Legacy hierarchy not writable, skipping
lxc-start simple-alpine-nested 20220904033500.390 INFO cgfsng - cgroups/cgfsng.c:legacy_hierarchy_delegated:3060 - Permission denied - Legacy hierarchy not writable, skipping
lxc-start simple-alpine-nested 20220904033500.390 INFO cgfsng - cgroups/cgfsng.c:unified_hierarchy_delegated:3047 - Permission denied - The cgroup.threads file is not writable, skipping unified hierarchy
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "reject_force_umount # comment this to allow umount -f; not recommended"
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "[all]"
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "kexec_load errno 1"
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[246:kexec_load] action[327681:errno] arch[0]
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741827]
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741886]
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "open_by_handle_at errno 1"
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[304:open_by_handle_at] action[327681:errno] arch[0]
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741827]
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741886]
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "init_module errno 1"
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[175:init_module] action[327681:errno] arch[0]
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741827]
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741886]
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "finit_module errno 1"
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[313:finit_module] action[327681:errno] arch[0]
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741827]
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741886]
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "delete_module errno 1"
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[176:delete_module] action[327681:errno] arch[0]
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741827]
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741886]
lxc-start simple-alpine-nested 20220904033500.390 INFO seccomp - seccomp.c:parse_config_v2:1017 - Merging compat seccomp contexts into main context
lxc-start simple-alpine-nested 20220904033500.390 INFO start - start.c:lxc_init:884 - Container "simple-alpine-nested" is initialized
lxc-start simple-alpine-nested 20220904033500.391 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWUSER
lxc-start simple-alpine-nested 20220904033500.391 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWNS
lxc-start simple-alpine-nested 20220904033500.391 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWPID
lxc-start simple-alpine-nested 20220904033500.391 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWUTS
lxc-start simple-alpine-nested 20220904033500.391 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWIPC
lxc-start simple-alpine-nested 20220904033500.391 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWCGROUP
lxc-start simple-alpine-nested 20220904033500.394 INFO start - start.c:do_start:1107 - Unshared CLONE_NEWNET
lxc-start simple-alpine-nested 20220904033500.394 NOTICE utils - utils.c:lxc_drop_groups:1365 - Dropped supplimentary groups
lxc-start simple-alpine-nested 20220904033500.394 NOTICE utils - utils.c:lxc_switch_uid_gid:1341 - Switched to gid 0
lxc-start simple-alpine-nested 20220904033500.394 NOTICE utils - utils.c:lxc_switch_uid_gid:1350 - Switched to uid 0
lxc-start simple-alpine-nested 20220904033500.395 WARN start - start.c:lxc_spawn:1835 - Operation not permitted - Failed to allocate new network namespace id
lxc-start simple-alpine-nested 20220904033500.395 INFO conf - conf.c:setup_utsname:875 - Set hostname to "simple-alpine-nested"
lxc-start simple-alpine-nested 20220904033500.395 INFO conf - conf.c:mount_autodev:1215 - Preparing "/dev"
lxc-start simple-alpine-nested 20220904033500.395 INFO conf - conf.c:mount_autodev:1276 - Prepared "/dev"
lxc-start simple-alpine-nested 20220904033500.396 ERROR utils - utils.c:safe_mount:1218 - Resource busy - Failed to mount "sys" onto "/usr/lib/lxc/rootfs/dev/.lxc/sys"
lxc-start simple-alpine-nested 20220904033500.396 INFO conf - conf.c:mount_entry:2403 - Resource busy - Failed to mount "sys" on "/usr/lib/lxc/rootfs/dev/.lxc/sys" (optional)
lxc-start simple-alpine-nested 20220904033500.396 INFO conf - conf.c:run_script_argv:337 - Executing script "/usr/share/lxcfs/lxc.mount.hook" for container "simple-alpine-nested", config section "lxc"
lxc-start simple-alpine-nested 20220904033500.427 INFO conf - conf.c:lxc_fill_autodev:1313 - Populating "/dev"
lxc-start simple-alpine-nested 20220904033500.428 INFO conf - conf.c:lxc_fill_autodev:1401 - Populated "/dev"
lxc-start simple-alpine-nested 20220904033500.428 INFO conf - conf.c:lxc_transient_proc:3771 - Caller's PID is 1; /proc/self points to 1
lxc-start simple-alpine-nested 20220904033500.428 INFO conf - conf.c:lxc_allocate_ttys:1106 - Finished creating 4 tty devices
lxc-start simple-alpine-nested 20220904033500.428 INFO conf - conf.c:lxc_setup_ttys:1072 - Finished setting up 4 /dev/tty<N> device(s)
lxc-start simple-alpine-nested 20220904033500.429 INFO conf - conf.c:setup_personality:1913 - Set personality to "0lx0"
lxc-start simple-alpine-nested 20220904033500.429 NOTICE conf - conf.c:lxc_setup:4464 - The container "simple-alpine-nested" is set up
lxc-start simple-alpine-nested 20220904033500.429 NOTICE start - start.c:start:2161 - Exec'ing "/sbin/init"
lxc-start simple-alpine-nested 20220904033500.430 NOTICE start - start.c:post_start:2172 - Started "/sbin/init" with pid "22898"
lxc-start simple-alpine-nested 20220904033500.430 NOTICE start - start.c:signal_handler:449 - Received 17 from pid 22899 instead of container init 22898
localhost:~$
When I try to create similar unprivileged container as root, /sys/fs/cgroups are populated, attempting to start docker service inside the container produces less errors, and docker run hello-world works.
So this must be somehow related to me running the unprivileged container as an user, not as a root.