Unable to load properly LXC without systemd service

Hi,

I’m working on a mips embedded platform equipped with kernel linux 3.10.104 and I’ve installed on my firmware linux distro lxc 4.0.12 successfully.
So later I’ve enabled cgroups linux configs, below you have lxc-checkconfig printout and lxc container test config.
Unfortunately I can’t start with LXC API a valid container, instead if I use lxc commands with bash shell (user root) the container will be created well.
What should I modify in order to get a correct container setup? Is there anything related to cgroups which fails?

Best regards,
Francesco

lxc-checkconfig
LXC version 4.0.12
— Namespaces —
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
newuidmap is not installed
newgidmap is not installed
Network namespace: enabled
Multiple /dev/pts instances: enabled

— Control groups —
Cgroups: enabled
Cgroup namespace: missing

Cgroup v1 mount points:
/sys/fs/cgroup/cpuset
/sys/fs/cgroup/cpu
/sys/fs/cgroup/cpuacct
/sys/fs/cgroup/memory
/sys/fs/cgroup/devices
/sys/fs/cgroup/freezer

Cgroup v2 mount points:

Cgroup v1 systemd controller: missing
Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

— Misc —
Veth pair device: enabled, not loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, not loaded
Advanced netfilter: enabled, not loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, not loaded

— Checkpoint/Restore —
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities:

I noticed that cgroup v1 systemd controller hasn’t been enabled but I don’t care since on my embedded system I don’t work with systemd framework.

/var/lib/lxc/test/config
lxc.uts.name = test
lxc.rootfs.path = /mnt/DiskE/test
lxc.init.cmd = /sbin/init
lxc.log.level = 0
lxc.log.syslog = daemon
lxc.rootfs.managed = 0
lxc.autodev = 1
lxc.tty.max = 5
lxc.pty.max = 1
lxc.mount.auto = cgroup proc:mixed sys:ro
lxc.mount.entry = shm dev/shm tmpfs defaults,create=dir 0 0
lxc.mount.entry = mqueue dev/mqueue mqueue defaults,optional,create=dir 0 0
lxc.mount.entry = /bin bin none ro,bind 0 0
lxc.mount.entry = /lib lib none ro,bind 0 0

Here you have the printout about the container startup with API functions:

lxc[2994]: test: start - start.c:lxc_start:2188 - Doing lxc_start
lxc[2994]: test: lsm - lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver nop
lxc[2994]: test: start - start.c:lxc_init:781 - Initialized LSM
lxc[2994]: test: start - start.c:lxc_serve_state_clients:486 - Set container state to STARTING
lxc[2994]: test: start - start.c:lxc_serve_state_clients:489 - No state clients registered
lxc[2994]: test: start - start.c:lxc_init:787 - Set container state to “STARTING”
lxc[2994]: test: start - start.c:lxc_init:843 - Set environment variables
lxc[2994]: test: start - start.c:lxc_init:848 - Ran pre-start hooks
lxc[2994]: test: terminal - terminal.c:lxc_devpts_terminal:992 - Inappropriate ioctl for device - Pure fd-based terminal allocation not possible
lxc[2994]: test: start - start.c:setup_signal_fd:373 - Created signal file descriptor 6
lxc[2994]: test: start - start.c:lxc_init:861 - Set up signal fd
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgroup_hierarchy_add:457 - Adding cgroup hierarchy mounted at freezer and base cgroup (null)
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgroup_hierarchy_add:459 - The hierarchy contains the freezer controller
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgroup_hierarchy_add:457 - Adding cgroup hierarchy mounted at devices and base cgroup (null)
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgroup_hierarchy_add:459 - The hierarchy contains the devices controller
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgroup_hierarchy_add:457 - Adding cgroup hierarchy mounted at memory and base cgroup (null)
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgroup_hierarchy_add:459 - The hierarchy contains the memory controller
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgroup_hierarchy_add:457 - Adding cgroup hierarchy mounted at cpuacct and base cgroup (null)
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgroup_hierarchy_add:459 - The hierarchy contains the cpuacct controller
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgroup_hierarchy_add:457 - Adding cgroup hierarchy mounted at cpu and base cgroup (null)
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgroup_hierarchy_add:459 - The hierarchy contains the cpu controller
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgroup_hierarchy_add:457 - Adding cgroup hierarchy mounted at cpuset and base cgroup (null)
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgroup_hierarchy_add:459 - The hierarchy contains the cpuset controller
lxc[2994]: test: cgroup - cgroups/cgroup.c:cgroup_init:40 - Initialized cgroup driver cgfsng
lxc[2994]: test: cgroup - cgroups/cgroup.c:cgroup_init:43 - Legacy cgroup layout
lxc[2994]: test: start - start.c:lxc_init:868 - Initialized cgroup driver
lxc[2994]: test: start - start.c:lxc_init:875 - Read seccomp policy
lxc[2994]: test: start - start.c:lxc_init:882 - Initialized LSM
lxc[2994]: test: start - start.c:lxc_init:884 - Container “test” is initialized
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:715 - Created 9(lxc.monitor.test) cgroup
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:715 - Created 10(lxc.monitor.test) cgroup
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:715 - Created 11(lxc.monitor.test) cgroup
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:715 - Created 12(lxc.monitor.test) cgroup
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:715 - Created 13(lxc.monitor.test) cgroup
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:715 - Created 14(lxc.monitor.test) cgroup
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cpuset1_cpus_initialize:595 - The path “/sys/devices/system/cpu/isolated” to read isolated cpus from does not exist
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cpuset1_cpus_initialize:626 - No isolated or offline cpus present in cpuset
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cpuset1_cpus_initialize:640 - Copied cpu settings of parent cgroup
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cpuset1_initialize:676 - Initialized cpuset in the legacy hierarchy
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1028 - The monitor process uses “lxc.monitor.test” as cgroup
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_enter:1178 - Moved monitor into cgroup 15
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_enter:1187 - Moved transient process into cgroup 15
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_enter:1178 - Moved monitor into cgroup 16
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_enter:1187 - Moved transient process into cgroup 16
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_enter:1178 - Moved monitor into cgroup 17
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_enter:1187 - Moved transient process into cgroup 17
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_enter:1178 - Moved monitor into cgroup 18
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_enter:1187 - Moved transient process into cgroup 18
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_enter:1178 - Moved monitor into cgroup 19
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_enter:1185 - Invalid argument - Failed to enter cgroup 19
lxc[2994]: test: start - start.c:__lxc_start:2035 - Failed to enter monitor cgroup
lxc[2994]: test: start - start.c:lxc_serve_state_socket_pair:548 - Sent container state “ABORTING” to 90
lxc[2994]: test: start - start.c:lxc_serve_state_clients:486 - Set container state to ABORTING
lxc[2994]: test: start - start.c:lxc_serve_state_clients:489 - No state clients registered
lxc[2994]: test: start - start.c:lxc_serve_state_clients:486 - Set container state to STOPPING
lxc[2994]: test: start - start.c:lxc_serve_state_clients:489 - No state clients registered
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_payload_destroy:548 - Uninitialized limit cgroup
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:715 - Created 9(lxc.pivot) cgroup
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_destroy:913 - Device or resource busy - Failed to destroy 9(lxc.monitor.test)
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:715 - Created 10(lxc.pivot) cgroup
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_destroy:913 - Device or resource busy - Failed to destroy 10(lxc.monitor.test)
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:715 - Created 11(lxc.pivot) cgroup
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_destroy:913 - Device or resource busy - Failed to destroy 11(lxc.monitor.test)
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:715 - Created 12(lxc.pivot) cgroup
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_destroy:913 - Device or resource busy - Failed to destroy 12(lxc.monitor.test)
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:715 - Created 13(lxc.pivot) cgroup
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_destroy:915 - Removed cgroup tree 13(lxc.monitor.test)
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:715 - Created 14(lxc.pivot) cgroup
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cpuset1_cpus_initialize:595 - The path “/sys/devices/system/cpu/isolated” to read isolated cpus from does not exist
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cpuset1_cpus_initialize:626 - No isolated or offline cpus present in cpuset
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cpuset1_cpus_initialize:640 - Copied cpu settings of parent cgroup
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cpuset1_initialize:676 - Initialized cpuset in the legacy hierarchy
lxc[2994]: test: cgfsng - cgroups/cgfsng.c:cgfsng_monitor_destroy:915 - Removed cgroup tree 14(lxc.monitor.test)
lxc[2994]: test: start - start.c:lxc_end:966 - Closed command socket
lxc[2994]: test: start - start.c:lxc_end:977 - Set container state to “STOPPED”

So to clarify on the same system, you have a container which you can lxc-start -n NAME just fine but can’t perform the same using your own program using liblxc?

Exactly.

Even when you’re running both lxc-start and your equivalent from the same terminal?

No, when I start the equivalent program from the terminal I can create and run LXC container correctly, matching the same beaviour as lxc-start application.

What’s running your program in the failing environment? Some kind of init system?

Yes, there’s a linux distribution where the system runs init as busybox applet.

Ok, what you’d need to figure out is what’s different between your init system calling your binary and you calling your binary from a shell after boot.

Maybe it’s just a timing thing and it’s running too early?

Trying to apply some syncronous delay to startup code but it doesn’t fix this issue. Now I’ve applied a workaround so I call lxc-start application within my firmware code and it works.