LXC opensuse unprivileged container

seven · March 25, 2019, 12:37pm

@stgraber @Jeremy Hello, today i revisited it, i somehow got it working; an issue though, i was trying to limit cpu, but i got the following error;

lxc-start -n my-cont -d -F
lxc-start: cgroups/cgfsng.c: lxc_cgroup_set_data: 1931 Failed to setup limits for the "cpuset" controller. The controller seems to be unused by "cgfsng" cgroup driver or not enabled on the cgroup hierarchy
                                                                                                  lxc-start: cgroups/cgfsng.c: cgfsng_setup_limits: 1968 No such file or directory - Error setting cpuset.cpus to 0 for my-cont
             lxc-start: start.c: lxc_spawn: 1356 Failed to setup cgroup limits for container "my-cont".
                                                                                                      lxc-start: start.c: __lxc_start: 1530 Failed to spawn container "my-cont".
                                                                       lxc-start: tools/lxc_start.c: main: 368 The container failed to start.
lxc-start: tools/lxc_start.c: main: 372 Additional information can be obtained by setting the --logfile and --logpriority options.

for more details on this, i filed an issue here;

github.com/lxc/lxc

Can not limit resources.

opened 06:49AM - 25 Mar 19 UTC

closed 01:56PM - 20 Mar 20 UTC

uwejan

The template below is mostly useful for bug reports and support questions. Feel… free to remove anything which doesn't apply to you and add more information where it makes sense. # Required information Linux OpenSuse 15 ``` lxc-start --version 2.0.9 lxc-checkconfig --- Namespaces --- Namespaces: enabled Utsname namespace: enabled Ipc namespace: enabled Pid namespace: enabled User namespace: enabled Network namespace: enabled --- Control groups --- Cgroups: enabled Cgroup v1 mount points: /sys/fs/cgroup/systemd /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/perf_event /sys/fs/cgroup/devices /sys/fs/cgroup/cpuset /sys/fs/cgroup/net_cls,net_prio /sys/fs/cgroup/blkio /sys/fs/cgroup/freezer /sys/fs/cgroup/rdma /sys/fs/cgroup/hugetlb /sys/fs/cgroup/pids /sys/fs/cgroup/memory Cgroup v2 mount points: /sys/fs/cgroup/unified Cgroup v1 clone_children flag: enabled Cgroup device: enabled Cgroup sched: enabled Cgroup cpu account: enabled Cgroup memory controller: enabled Cgroup cpuset: enabled --- Misc --- Veth pair device: enabled, not loaded Macvlan: enabled, loaded Vlan: enabled, not loaded Bridges: enabled, loaded Advanced netfilter: enabled, not loaded CONFIG_NF_NAT_IPV4: enabled, loaded CONFIG_NF_NAT_IPV6: enabled, loaded CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loadedCONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded FUSE (for use with lxcfs): enabled, loaded --- Checkpoint/Restore --- checkpoint restore: enabled CONFIG_FHANDLE: enabled CONFIG_EVENTFD: enabled CONFIG_EPOLL: enabled CONFIG_UNIX_DIAG: enabled CONFIG_INET_DIAG: enabled CONFIG_PACKET_DIAG: enabled CONFIG_NETLINK_DIAG: enabled File capabilities: enabled Note : Before booting a new kernel, you can check its configuration usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig uname -a Linux static 4.12.14-lp150.12.48-default cat /proc/self/cgroup 12:memory:/user/lxc/0 11:pids:/user.slice/user-1000.slice/session-2.scope 10:hugetlb:/ 9:rdma:/ 8:freezer:/user/lxc/0 7:blkio:/user.slice 6:net_cls,net_prio:/ 5:cpuset:/ 4:devices:/user.slice 3:perf_event:/ 2:cpu,cpuacct:/user.slice 1:name=systemd:/user/lxc/0 0::/user.slice/user-1000.slice/session-2.scope cat /proc/1/mounts sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,nosuid,size=32858256k,nr_inodes=8214564,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset,clone_children 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 /dev/nvme0n1p3 / ext4 rw,relatime,data=ordered 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=666 0 0 debugfs /sys/kernel/debug debugfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0 /dev/nvme1n1p1 /home ext4 rw,relatime,data=ordered 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=6573276k,mode=700,uid=1000,gid=100 0 0 lxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0 ``` # Issue description Am trying to limit cpu usage to 1 cpu, i think its called cpu mapping unlike what i was looking for which use `just use 1 cpu without specifying exactly which 1 to use` so in the container config file i added the following ; `lxc.cgroup.cpuset.cpus = 0` when starting the container i get the flowing error. ``` lxc-start -n my-cont -d -F lxc-start: cgroups/cgfsng.c: lxc_cgroup_set_data: 1931 Failed to setup limits for the "cpuset" controller. The controller seems to be unused by "cgfsng" cgroup driver or not enabled on the cgroup hierarchy lxc-start: cgroups/cgfsng.c: cgfsng_setup_limits: 1968 No such file or directory - Error setting cpuset.cpus to 0 for my-cont lxc-start: start.c: lxc_spawn: 1356 Failed to setup cgroup limits for container "my-cont". lxc-start: start.c: __lxc_start: 1530 Failed to spawn container "my-cont". lxc-start: tools/lxc_start.c: main: 368 The container failed to start. lxc-start: tools/lxc_start.c: main: 372 Additional information can be obtained by setting the --logfile and --logpriority options. ``` How do i resolve this please? second question where do i find all possible configs of limiting resources? are they listed somewhere? # Information to attach ``` findmnt | grep cgroup │ ├─/sys/fs/cgroup tmpfs tmpfs ro,nosuid,nodev,noexec,mode=755 │ │ ├─/sys/fs/cgroup/unified cgroup cgroup2 rw,nosuid,nodev,noexec,relatime │ │ ├─/sys/fs/cgroup/systemd cgroup cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd │ │ ├─/sys/fs/cgroup/cpu,cpuacct cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct │ │ ├─/sys/fs/cgroup/perf_event cgroup cgroup rw,nosuid,nodev,noexec,relatime,perf_event │ │ ├─/sys/fs/cgroup/devices cgroup cgroup rw,nosuid,nodev,noexec,relatime,devices │ │ ├─/sys/fs/cgroup/cpuset cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpuset,clone_children │ │ ├─/sys/fs/cgroup/net_cls,net_prio cgroup cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio │ │ ├─/sys/fs/cgroup/blkio cgroup cgroup rw,nosuid,nodev,noexec,relatime,blkio │ │ ├─/sys/fs/cgroup/freezer cgroup cgroup rw,nosuid,nodev,noexec,relatime,freezer │ │ ├─/sys/fs/cgroup/rdma cgroup cgroup rw,nosuid,nodev,noexec,relatime,rdma │ │ ├─/sys/fs/cgroup/hugetlb cgroup cgroup rw,nosuid,nodev,noexec,relatime,hugetlb │ │ ├─/sys/fs/cgroup/pids cgroup cgroup rw,nosuid,nodev,noexec,relatime,pids │ │ └─/sys/fs/cgroup/memory cgroup cgroup rw,nosuid,nodev,noexec,relatime,memory ```

stgraber · March 25, 2019, 7:57pm

Sounds like your kernel is missing that cgroup controller or your host doesn’t have it mounted?

seven · March 25, 2019, 9:19pm

@stgraber in either cases, what would i do please?

stgraber · March 27, 2019, 4:54pm

Either make the cpuset controller work on your system or don’t use any cpuset.cpus limits as those require that cgroup.

seven · March 28, 2019, 9:39am

@stgraber i get it, but if i knew how to make it work, i would not be posting here. can you help fixing the issue please?

stgraber · March 28, 2019, 2:33pm

I don’t have any opensuse knowledge so can’t help much there, was just trying to give you some pointers so you can hopefully do some googling and find someone who explains how to do this on your distro.

The second option may be easier for you though, just don’t use the cpuset cgroup.
Edit your container’s config and remove any line that reference lxc.cgroup.cpuset (looks like you have lxc.cgroup.cpuset.cpus=0) and your container should start fine.

seven · March 28, 2019, 3:57pm

@stgraber i understand, i been googling since i got the issue, can not find any solution, rather its more ubuntu/debian related and the only source of info is this forum or the github issues section, disabling cpu limit/ memory/swap is not an option, if i was to use lxc instead of kvm i need to be doing it right.
imagine i had ubuntu instead of suse, what will you do to fix this issue, change permission of something? add code of line somewhere? just something that can really help solving it, if it works on ubuntu it has to work on other linux destro, its just the matter of knowing what to modify, i think.

seven · March 29, 2019, 7:14am

@stgraber , i kept experimenting and i got the cpu to be limited to core 0 i think, i had to add cpuset,cpu,cpuacct to my common-sesion-pc, as follows;
session optional pam_cgfs.so -c freezer,memory,cpuset,cpu,cpuacct,
i do not know what dose that mean, adding this lines, can you explain it please? plus am unable to have limit for swaplxc-start: cgroups/cgfsng.c: cgfsng_setup_limits: 1968 Permission denied - Error setting memory.memsw.limit_in_bytes to 1G for my-cont, last question, where do i find all this limits or options, are they listed in some web-page?

seven · March 29, 2019, 8:07am

even container start, i kept debugging using a file, here is some of its contents;

      lxc-start 20190329080105.142 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
      lxc-start 20190329080105.142 INFO     lxc_utils - utils.c:get_rundir:284 - XDG_RUNTIME_DIR isn't set in the environment.
      lxc-start 20190329080105.142 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
      lxc-start 20190329080105.142 DEBUG    lxc_start - start.c:setup_signal_fd:288 - Set SIGCHLD handler with file descriptor: 7.
      lxc-start 20190329080105.142 DEBUG    console - console.c:lxc_console_peer_default:450 - process does not have a controlling terminal
      lxc-start 20190329080105.142 DEBUG    lxc_conf - conf.c:chown_mapped_root:2830 - trying to chown "/dev/pts/7" to 1001
lxc-start 20190329080105.155 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2516 - The binary "/usr/bin/newuidmap" does have the setuid bit set.
      lxc-start 20190329080105.155 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2516 - The binary "/usr/bin/newgidmap" does have the setuid bit set.
      lxc-start 20190329080105.155 DEBUG    lxc_conf - conf.c:lxc_map_ids:2604 - Functional newuidmap and newgidmap binary found.
lxc-start 20190329080105.165 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1971 - cgroup 'cpuset.cpus' set to '0'
      lxc-start 20190329080105.165 INFO     lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1976 - cgroup has been setup
      lxc-start 20190329080105.166 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2516 - The binary "/usr/bin/newuidmap" does have the setuid bit set.
      lxc-start 20190329080105.166 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2516 - The binary "/usr/bin/newgidmap" does have the setuid bit set.
      lxc-start 20190329080105.166 DEBUG    lxc_conf - conf.c:lxc_map_ids:2604 - Functional newuidmap and newgidmap binary found.
      lxc-start 20190329080105.181 WARN     lxc_cgfsng - cgroups/cgfsng.c:chown_cgroup_wrapper:1465 - Error chmoding /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-1.scope/user/lxc/0/lxc/kaltura: No such file or directory
      lxc-start 20190329080105.183 INFO     lxc_network - network.c:lxc_create_network_unpriv_exec:2081 - Execing lxc-user-nic create /home/lxc/.local/share/lxc kaltura 4044 veth br0 (null)
      lxc-start 20190329080105.289 NOTICE   lxc_utils - utils.c:lxc_switch_uid_gid:2070 - Switched to gid 0.
      lxc-start 20190329080105.289 NOTICE   lxc_utils - utils.c:lxc_switch_uid_gid:2076 - Switched to uid 0.
      lxc-start 20190329080105.289 NOTICE   lxc_utils - utils.c:lxc_setgroups:2088 - Dropped additional groups.
 lxc-start 20190329080105.350 INFO     lxc_conf - conf.c:lxc_fill_autodev:1165 - Populating "/dev"
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_fill_autodev:1209 - Bind mounted "/dev/null" onto "/usr/lib64/lxc/rootfs/dev/null"
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_fill_autodev:1209 - Bind mounted "/dev/zero" onto "/usr/lib64/lxc/rootfs/dev/zero"
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_fill_autodev:1209 - Bind mounted "/dev/full" onto "/usr/lib64/lxc/rootfs/dev/full"
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_fill_autodev:1209 - Bind mounted "/dev/urandom" onto "/usr/lib64/lxc/rootfs/dev/urandom"
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_fill_autodev:1209 - Bind mounted "/dev/random" onto "/usr/lib64/lxc/rootfs/dev/random"
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_fill_autodev:1209 - Bind mounted "/dev/tty" onto "/usr/lib64/lxc/rootfs/dev/tty"
      lxc-start 20190329080105.350 INFO     lxc_conf - conf.c:lxc_fill_autodev:1216 - Populated "/dev"
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_setup_dev_console:1537 - mounted pts device "/dev/pts/7" onto "/usr/lib64/lxc/rootfs/dev/console"
      lxc-start 20190329080105.350 INFO     lxc_utils - utils.c:lxc_mount_proc_if_needed:1755 - I am 1, /proc/self points to "1"
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:setup_rootfs_pivot_root:1068 - pivot_root syscall to '/usr/lib64/lxc/rootfs' successful
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:setup_pivot_root:1377 - finished pivot root
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_setup_devpts:1416 - mount new devpts instance with options "newinstance,ptmxmode=0666,mode=0620,gid=5"
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_setup_devpts:1436 - created dummy "/dev/ptmx" file as bind mount target
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_setup_devpts:1441 - bind mounted "/dev/pts/ptmx" to "/dev/ptmx"
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_allocate_ttys:905 - allocated pty "/dev/pts/0" with master fd 11 and slave fd 14
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_allocate_ttys:905 - allocated pty "/dev/pts/1" with master fd 15 and slave fd 16
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_allocate_ttys:905 - allocated pty "/dev/pts/2" with master fd 17 and slave fd 18
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_allocate_ttys:905 - allocated pty "/dev/pts/3" with master fd 19 and slave fd 20
      lxc-start 20190329080105.350 INFO     lxc_conf - conf.c:lxc_allocate_ttys:925 - finished allocating 4 pts devices
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_setup_ttys:862 - Bind mounted "/dev/pts/0" onto "/dev/tty1"
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_setup_ttys:862 - Bind mounted "/dev/pts/1" onto "/dev/tty2"
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_setup_ttys:862 - Bind mounted "/dev/pts/2" onto "/dev/tty3"
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:lxc_setup_ttys:862 - Bind mounted "/dev/pts/3" onto "/dev/tty4"
      lxc-start 20190329080105.350 INFO     lxc_conf - conf.c:lxc_setup_ttys:871 - Finished setting up 4 /dev/tty<N> device(s)
      lxc-start 20190329080105.350 INFO     lxc_conf - conf.c:setup_personality:1477 - set personality to '0x0'
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:setup_caps:2329 - drop capability 'mac_admin' (33)
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:setup_caps:2329 - drop capability 'mac_override' (32)
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:setup_caps:2329 - drop capability 'sys_time' (25)
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:setup_caps:2329 - drop capability 'sys_module' (16)
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:setup_caps:2329 - drop capability 'sys_rawio' (17)
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:setup_caps:2329 - drop capability 'sys_nice' (23)
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:setup_caps:2329 - drop capability 'sys_pacct' (20)
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:setup_caps:2329 - drop capability 'sys_rawio' (17)
      lxc-start 20190329080105.350 DEBUG    lxc_conf - conf.c:setup_caps:2338 - capabilities have been setup
      lxc-start 20190329080105.350 NOTICE   lxc_conf - conf.c:lxc_setup:3221 - Container "kaltura" is set up
      lxc-start 20190329080105.350 INFO     lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:238 - changed apparmor profile to lxc-container-default-cgns
      lxc-start 20190329080105.351 INFO     lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1976 - cgroup has been setup
      lxc-start 20190329080105.351 NOTICE   lxc_start - start.c:start:1603 - Exec'ing "/sbin/init".
      lxc-start 20190329080105.353 NOTICE   lxc_start - start.c:post_start:1614 - Started "/sbin/init" with pid "4044".
      lxc-start 20190329080105.353 INFO     lxc_utils - utils.c:get_rundir:284 - XDG_RUNTIME_DIR isn't set in the environment.
      lxc-start 20190329080105.353 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
      lxc-start 20190329080105.353 INFO     lxc_utils - utils.c:get_rundir:284 - XDG_RUNTIME_DIR isn't set in the environment.
      lxc-start 20190329080105.353 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
      lxc-start 20190329080105.353 NOTICE   lxc_start - start.c:signal_handler:337 - Received SIGCHLD from pid 4039 instead of container init 4044.

please have a look, dose any seem wrong/ requires to be fixed?

stgraber · March 29, 2019, 2:12pm

Nope, looks fine

seven · March 29, 2019, 4:16pm

@stgraber what would you say about post 28? Btw i do not seem to be able to autostart a container.