Hello,
I’m having trouble starting a container in unpriviledged mode.
Here is the output of lxc-checkconfig:
# lxc-checkconfig
LXC version 4.0.10
Kernel configuration not found at /proc/config.gz; searching...
Kernel configuration found at /boot/config-5.10.0-17-amd64
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
--- Control groups ---
Cgroups: enabled
Cgroup namespace: enabled
Cgroup v1 mount points:
Cgroup v2 mount points:
/sys/fs/cgroup
Cgroup v1 systemd controller: missing
Cgroup v1 freezer controller: missing
Cgroup ns_cgroup: required
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled
--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, loaded
CONFIG_NF_NAT_IPV4: missing
CONFIG_NF_NAT_IPV6: missing
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, loaded
--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities:
Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig
Here is the config of my container:
lxc.net.0.type = veth
lxc.net.0.link = br0
lxc.net.0.veth.pair = vMyCT
lxc.net.0.hwaddr = 00:FF:AA:BB:06:03
lxc.net.0.ipv4.address = 10.0.6.3/24
lxc.net.0.ipv4.gateway = 10.0.6.254
lxc.net.0.flags = up
lxc.net.0.name = eth0
## mount points
lxc.rootfs.options = defaults,ro,nodev
lxc.rootfs.path = dir:/mnt/ct_mount
lxc.hook.pre-mount = /sbin/base_myct_hook.sh
lxc.mount.fstab = /opt/CTs/MYCT.CT/fstab
# Common configuration
lxc.include = /usr/share/lxc/config/mylxc.common.conf
# Container specific configuration
lxc.uts.name = MYCT.CT
lxc.arch = amd64
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
# dropping capa linux
lxc.cap.drop =
lxc.cap.keep = setuid setgid sys_chroot net_bind_service chown audit_write dac_override
The configuration for the root inside /etc/subuid and /etc/subgid is done.
The hook script is basically a mount of a squashfs on a loop device.
#!/bin/bash
mount -o loop,ro -t squashfs /opt/BASES/BASE_MYCT.sqsh ${LXC_ROOTFS_PATH}
The squash is an image of a minimalist debian 11 made with debootstrap.
At first I had this output from lxc-start MYCT.CT -o /tmp/myct.ct.log -l debug
lxc-start MYCT.CT 20220913094837.638 INFO confile - confile.c:set_config_idmaps:2115 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start MYCT.CT 20220913094837.638 INFO confile - confile.c:set_config_idmaps:2115 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start MYCT.CT 20220913094837.638 INFO lxccontainer - lxccontainer.c:do_lxcapi_start:988 - Set process title to [lxc monitor] /opt/CTs MYCT.CT
lxc-start MYCT.CT 20220913094837.638 DEBUG lxccontainer - lxccontainer.c:wait_on_daemonized_start:849 - First child 6108 exited
lxc-start MYCT.CT 20220913094837.638 INFO lsm - lsm/lsm.c:lsm_init_static:40 - Initialized LSM security driver nop
lxc-start MYCT.CT 20220913094837.639 DEBUG terminal - terminal.c:lxc_terminal_peer_default:665 - No such device - The process does not have a controlling terminal
lxc-start MYCT.CT 20220913094837.639 ERROR cgroup2_devices - cgroups/cgroup2_devices.c:bpf_program_load_kernel:334 - Operation not permitted - Failed to load bpf program: (null)
lxc-start MYCT.CT 20220913094837.639 INFO start - start.c:lxc_init:889 - Container "MYCT.CT" is initialized
lxc-start MYCT.CT 20220913094837.668 INFO cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1070 - The monitor process uses "lxc.monitor.MYCT.CT" as cgroup
lxc-start MYCT.CT 20220913094837.669 DEBUG storage - storage/storage.c:get_storage_by_name:211 - Detected rootfs type "dir"
lxc-start MYCT.CT 20220913094837.669 DEBUG storage - storage/storage.c:get_storage_by_name:211 - Detected rootfs type "dir"
lxc-start MYCT.CT 20220913094837.669 INFO cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1178 - The container process uses "lxc.payload.MYCT.CT" as inner and "lxc.payload.MYCT.CT" as limit cgroup
lxc-start MYCT.CT 20220913094837.669 INFO start - start.c:lxc_spawn:1748 - Cloned CLONE_NEWUSER
lxc-start MYCT.CT 20220913094837.669 INFO start - start.c:lxc_spawn:1748 - Cloned CLONE_NEWNS
lxc-start MYCT.CT 20220913094837.669 INFO start - start.c:lxc_spawn:1748 - Cloned CLONE_NEWPID
lxc-start MYCT.CT 20220913094837.669 INFO start - start.c:lxc_spawn:1748 - Cloned CLONE_NEWUTS
lxc-start MYCT.CT 20220913094837.669 INFO start - start.c:lxc_spawn:1748 - Cloned CLONE_NEWIPC
lxc-start MYCT.CT 20220913094837.669 INFO start - start.c:lxc_spawn:1748 - Cloned CLONE_NEWCGROUP
lxc-start MYCT.CT 20220913094837.669 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved user namespace via fd 20 and stashed path as user:/proc/6109/fd/20
lxc-start MYCT.CT 20220913094837.669 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved mnt namespace via fd 21 and stashed path as mnt:/proc/6109/fd/21
lxc-start MYCT.CT 20220913094837.669 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved pid namespace via fd 22 and stashed path as pid:/proc/6109/fd/22
lxc-start MYCT.CT 20220913094837.669 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved uts namespace via fd 23 and stashed path as uts:/proc/6109/fd/23
lxc-start MYCT.CT 20220913094837.669 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved ipc namespace via fd 24 and stashed path as ipc:/proc/6109/fd/24
lxc-start MYCT.CT 20220913094837.669 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved cgroup namespace via fd 25 and stashed path as cgroup:/proc/6109/fd/25
lxc-start MYCT.CT 20220913094837.669 DEBUG conf - conf.c:idmaptool_on_path_and_privileged:3317 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start MYCT.CT 20220913094837.669 DEBUG conf - conf.c:idmaptool_on_path_and_privileged:3317 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start MYCT.CT 20220913094837.669 DEBUG conf - conf.c:lxc_map_ids:3402 - Functional newuidmap and newgidmap binary found
lxc-start MYCT.CT 20220913094837.674 WARN cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits_legacy:2746 - Invalid argument - Ignoring legacy cgroup limits on pure cgroup2 system
lxc-start MYCT.CT 20220913094837.674 DEBUG conf - conf.c:idmaptool_on_path_and_privileged:3317 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start MYCT.CT 20220913094837.674 DEBUG conf - conf.c:idmaptool_on_path_and_privileged:3317 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start MYCT.CT 20220913094837.674 INFO conf - conf.c:lxc_map_ids:3400 - Caller maps host root. Writing mapping directly
lxc-start MYCT.CT 20220913094837.674 NOTICE utils - utils.c:lxc_drop_groups:1345 - Dropped supplimentary groups
lxc-start MYCT.CT 20220913094837.675 INFO start - start.c:do_start:1119 - Unshared CLONE_NEWNET
lxc-start MYCT.CT 20220913094837.675 NOTICE utils - utils.c:lxc_drop_groups:1345 - Dropped supplimentary groups
lxc-start MYCT.CT 20220913094837.675 NOTICE utils - utils.c:lxc_switch_uid_gid:1321 - Switched to gid 0
lxc-start MYCT.CT 20220913094837.675 NOTICE utils - utils.c:lxc_switch_uid_gid:1330 - Switched to uid 0
lxc-start MYCT.CT 20220913094837.675 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved net namespace via fd 7 and stashed path as net:/proc/6109/fd/7
lxc-start MYCT.CT 20220913094837.678 INFO network - network.c:netdev_configure_server_veth:653 - Retrieved mtu 1500 from brSSH
lxc-start MYCT.CT 20220913094837.678 INFO network - network.c:netdev_configure_server_veth:718 - Attached "vADMIN_LOCAL" to bridge "brSSH"
lxc-start MYCT.CT 20220913094837.678 DEBUG network - network.c:netdev_configure_server_veth:849 - Instantiated veth tunnel "vADMIN_LOCAL <--> veth92jWmk"
lxc-start MYCT.CT 20220913094837.678 INFO conf - conf.c:run_script_argv:339 - Executing script "/sbin/base_admin_local_hook.sh" for container "MYCT.CT", config section "lxc"
lxc-start MYCT.CT 20220913094837.685 DEBUG conf - conf.c:run_buffer:312 - Script exec /sbin/base_admin_local_hook.sh MYCT.CT lxc pre-mount produced output: mount:
lxc-start MYCT.CT 20220913094837.685 DEBUG conf - conf.c:run_buffer:312 - Script exec /sbin/base_admin_local_hook.sh MYCT.CT lxc pre-mount produced output: /mnt/ct_mount: failed to setup loop device for /opt/BASES/BASE_ADMIN_LOCAL.sqsh.
lxc-start MYCT.CT 20220913094837.685 ERROR conf - conf.c:run_buffer:323 - Script exited with status 32
lxc-start MYCT.CT 20220913094837.685 ERROR conf - conf.c:lxc_setup_rootfs_prepare_root:3765 - Failed to run pre-mount hooks
lxc-start MYCT.CT 20220913094837.685 ERROR conf - conf.c:lxc_setup:4058 - Failed to setup rootfs
lxc-start MYCT.CT 20220913094837.685 ERROR start - start.c:do_start:1291 - Failed to setup container "MYCT.CT"
lxc-start MYCT.CT 20220913094837.685 ERROR sync - sync.c:sync_wait:36 - An error occurred in another process (expected sequence number 3)
lxc-start MYCT.CT 20220913094837.685 DEBUG network - network.c:lxc_delete_network:4185 - Deleted network devices
lxc-start MYCT.CT 20220913094837.685 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:868 - Received container state "ABORTING" instead of "RUNNING"
lxc-start MYCT.CT 20220913094837.685 ERROR lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start MYCT.CT 20220913094837.685 ERROR start - start.c:__lxc_start:2053 - Failed to spawn container "MYCT.CT"
lxc-start MYCT.CT 20220913094837.685 WARN start - start.c:lxc_abort:1050 - No such process - Failed to send SIGKILL via pidfd 19 for process 6118
lxc-start MYCT.CT 20220913094837.685 ERROR lxc_start - tools/lxc_start.c:main:311 - To get more details, run the container in foreground mode
lxc-start MYCT.CT 20220913094837.685 ERROR lxc_start - tools/lxc_start.c:main:313 - Additional information can be obtained by setting the --logfile and --logpriority options
lxc-start MYCT.CT 20220913094837.686 DEBUG conf - conf.c:idmaptool_on_path_and_privileged:3317 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start MYCT.CT 20220913094837.686 DEBUG conf - conf.c:idmaptool_on_path_and_privileged:3317 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start MYCT.CT 20220913094837.686 INFO conf - conf.c:lxc_map_ids:3400 - Caller maps host root. Writing mapping directly
lxc-start MYCT.CT 20220913094837.686 NOTICE utils - utils.c:lxc_drop_groups:1345 - Dropped supplimentary groups
lxc-start MYCT.CT 20220913094837.687 INFO conf - conf.c:run_script_argv:339 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "MYCT.CT", config section "lxc"
Using strace on the mount, I found that I was refused by the kernel loop driver (ioctl call with LOOP_GET_STATUS64 returning Operation not permitted).
I modified my hook script to use pkexec and got this output:
lxc-start MYCT.CT 20220913084917.447 INFO confile - confile.c:set_config_idmaps:2115 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start MYCT.CT 20220913084917.447 INFO confile - confile.c:set_config_idmaps:2115 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start MYCT.CT 20220913084917.448 INFO lxccontainer - lxccontainer.c:do_lxcapi_start:988 - Set process title to [lxc monitor] /opt/CTs MYCT.CT
lxc-start MYCT.CT 20220913084917.448 DEBUG lxccontainer - lxccontainer.c:wait_on_daemonized_start:849 - First child 12845 exited
lxc-start MYCT.CT 20220913084917.448 INFO lsm - lsm/lsm.c:lsm_init_static:40 - Initialized LSM security driver nop
lxc-start MYCT.CT 20220913084917.448 DEBUG terminal - terminal.c:lxc_terminal_peer_default:665 - No such device - The process does not have a controlling terminal
lxc-start MYCT.CT 20220913084917.448 ERROR cgroup2_devices - cgroups/cgroup2_devices.c:bpf_program_load_kernel:334 - Operation not permitted - Failed to load bpf program: (null)
lxc-start MYCT.CT 20220913084917.448 INFO start - start.c:lxc_init:889 - Container "MYCT.CT" is initialized
lxc-start MYCT.CT 20220913084917.480 INFO cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1070 - The monitor process uses "lxc.monitor.MYCT.CT" as cgroup
lxc-start MYCT.CT 20220913084917.481 DEBUG storage - storage/storage.c:get_storage_by_name:211 - Detected rootfs type "dir"
lxc-start MYCT.CT 20220913084917.481 DEBUG storage - storage/storage.c:get_storage_by_name:211 - Detected rootfs type "dir"
lxc-start MYCT.CT 20220913084917.481 INFO cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1178 - The container process uses "lxc.payload.MYCT.CT" as inner and "lxc.payload.MYCT.CT" as limit cgroup
lxc-start MYCT.CT 20220913084917.481 INFO start - start.c:lxc_spawn:1748 - Cloned CLONE_NEWUSER
lxc-start MYCT.CT 20220913084917.481 INFO start - start.c:lxc_spawn:1748 - Cloned CLONE_NEWNS
lxc-start MYCT.CT 20220913084917.481 INFO start - start.c:lxc_spawn:1748 - Cloned CLONE_NEWPID
lxc-start MYCT.CT 20220913084917.481 INFO start - start.c:lxc_spawn:1748 - Cloned CLONE_NEWUTS
lxc-start MYCT.CT 20220913084917.481 INFO start - start.c:lxc_spawn:1748 - Cloned CLONE_NEWIPC
lxc-start MYCT.CT 20220913084917.481 INFO start - start.c:lxc_spawn:1748 - Cloned CLONE_NEWCGROUP
lxc-start MYCT.CT 20220913084917.481 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved user namespace via fd 20 and stashed path as user:/proc/12846/fd/20
lxc-start MYCT.CT 20220913084917.481 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved mnt namespace via fd 21 and stashed path as mnt:/proc/12846/fd/21
lxc-start MYCT.CT 20220913084917.481 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved pid namespace via fd 22 and stashed path as pid:/proc/12846/fd/22
lxc-start MYCT.CT 20220913084917.481 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved uts namespace via fd 23 and stashed path as uts:/proc/12846/fd/23
lxc-start MYCT.CT 20220913084917.481 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved ipc namespace via fd 24 and stashed path as ipc:/proc/12846/fd/24
lxc-start MYCT.CT 20220913084917.481 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved cgroup namespace via fd 25 and stashed path as cgroup:/proc/12846/fd/25
lxc-start MYCT.CT 20220913084917.481 DEBUG conf - conf.c:idmaptool_on_path_and_privileged:3317 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start MYCT.CT 20220913084917.481 DEBUG conf - conf.c:idmaptool_on_path_and_privileged:3317 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start MYCT.CT 20220913084917.481 DEBUG conf - conf.c:lxc_map_ids:3402 - Functional newuidmap and newgidmap binary found
lxc-start MYCT.CT 20220913084917.485 WARN cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits_legacy:2746 - Invalid argument - Ignoring legacy cgroup limits on pure cgroup2 system
lxc-start MYCT.CT 20220913084917.485 DEBUG conf - conf.c:idmaptool_on_path_and_privileged:3317 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start MYCT.CT 20220913084917.485 DEBUG conf - conf.c:idmaptool_on_path_and_privileged:3317 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start MYCT.CT 20220913084917.485 INFO conf - conf.c:lxc_map_ids:3400 - Caller maps host root. Writing mapping directly
lxc-start MYCT.CT 20220913084917.485 NOTICE utils - utils.c:lxc_drop_groups:1345 - Dropped supplimentary groups
lxc-start MYCT.CT 20220913084917.486 INFO start - start.c:do_start:1119 - Unshared CLONE_NEWNET
lxc-start MYCT.CT 20220913084917.486 NOTICE utils - utils.c:lxc_drop_groups:1345 - Dropped supplimentary groups
lxc-start MYCT.CT 20220913084917.486 NOTICE utils - utils.c:lxc_switch_uid_gid:1321 - Switched to gid 0
lxc-start MYCT.CT 20220913084917.486 NOTICE utils - utils.c:lxc_switch_uid_gid:1330 - Switched to uid 0
lxc-start MYCT.CT 20220913084917.486 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved net namespace via fd 7 and stashed path as net:/proc/12846/fd/7
lxc-start MYCT.CT 20220913084917.490 INFO network - network.c:netdev_configure_server_veth:653 - Retrieved mtu 1500 from brSSH
lxc-start MYCT.CT 20220913084917.491 INFO network - network.c:netdev_configure_server_veth:718 - Attached "vADMIN_LOCAL" to bridge "brSSH"
lxc-start MYCT.CT 20220913084917.491 DEBUG network - network.c:netdev_configure_server_veth:849 - Instantiated veth tunnel "vADMIN_LOCAL <--> veth2IsRAc"
lxc-start MYCT.CT 20220913084917.491 INFO conf - conf.c:run_script_argv:339 - Executing script "/sbin/base_admin_local_hook.sh" for container "MYCT.CT", config section "lxc"
lxc-start MYCT.CT 20220913084917.505 DEBUG conf - conf.c:run_buffer:312 - Script exec /sbin/base_admin_local_hook.sh MYCT.CT lxc pre-mount produced output: Error getting authority: Error initializing authority: Exhausted all available authentication mechanisms (tried: EXTERNAL) (available: EXTERNAL)
lxc-start MYCT.CT 20220913084917.506 ERROR conf - conf.c:run_buffer:323 - Script exited with status 127
lxc-start MYCT.CT 20220913084917.506 ERROR conf - conf.c:lxc_setup_rootfs_prepare_root:3765 - Failed to run pre-mount hooks
lxc-start MYCT.CT 20220913084917.506 ERROR conf - conf.c:lxc_setup:4058 - Failed to setup rootfs
lxc-start MYCT.CT 20220913084917.506 ERROR start - start.c:do_start:1291 - Failed to setup container "MYCT.CT"
lxc-start MYCT.CT 20220913084917.506 ERROR sync - sync.c:sync_wait:36 - An error occurred in another process (expected sequence number 3)
lxc-start MYCT.CT 20220913084917.506 DEBUG network - network.c:lxc_delete_network:4185 - Deleted network devices
lxc-start MYCT.CT 20220913084917.506 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:868 - Received container state "ABORTING" instead of "RUNNING"
lxc-start MYCT.CT 20220913084917.506 ERROR lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start MYCT.CT 20220913084917.506 ERROR lxc_start - tools/lxc_start.c:main:311 - To get more details, run the container in foreground mode
lxc-start MYCT.CT 20220913084917.506 ERROR lxc_start - tools/lxc_start.c:main:313 - Additional information can be obtained by setting the --logfile and --logpriority options
lxc-start MYCT.CT 20220913084917.507 ERROR start - start.c:__lxc_start:2053 - Failed to spawn container "MYCT.CT"
lxc-start MYCT.CT 20220913084917.507 WARN start - start.c:lxc_abort:1050 - No such process - Failed to send SIGKILL via pidfd 19 for process 12853
lxc-start MYCT.CT 20220913084917.507 DEBUG conf - conf.c:idmaptool_on_path_and_privileged:3317 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start MYCT.CT 20220913084917.507 DEBUG conf - conf.c:idmaptool_on_path_and_privileged:3317 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start MYCT.CT 20220913084917.507 INFO conf - conf.c:lxc_map_ids:3400 - Caller maps host root. Writing mapping directly
lxc-start MYCT.CT 20220913084917.507 NOTICE utils - utils.c:lxc_drop_groups:1345 - Dropped supplimentary groups
lxc-start MYCT.CT 20220913084917.508 INFO conf - conf.c:run_script_argv:339 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "MYCT.CT", config section "lxc"
So my questions are:
1/ Am I doing things the right way?
2/ What uid/gid is used by lxc when he is calling the pre-mount script?