Can not bring up more than 11 LXC containers on ubuntu22

I have created 30+ copies of a LXC container but for some reason i can not bring up more that 11 LXCs , is there any limitation i shoud be aware of ? Please see the attached log.
I can bring up any of the 11 LXCs from 30 created but the 12th one fail to come up.

Log

lxc-start dut4lan2 20220612230430.440 INFO lxccontainer - lxccontainer.c:do_lxcapi_start:997 - Set process title to [lxc monitor] /home/umtts/comcast/tsport dut4lan2
lxc-start dut4lan2 20220612230430.440 DEBUG lxccontainer - lxccontainer.c:wait_on_daemonized_start:858 - First child 14675 exited
lxc-start dut4lan2 20220612230430.441 INFO lsm - lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start dut4lan2 20220612230430.441 INFO start - start.c:lxc_init:884 - Container “dut4lan2” is initialized
lxc-start dut4lan2 20220612230430.460 INFO cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1029 - The monitor process uses “lxc.monitor.dut4lan2” as cgroup
lxc-start dut4lan2 20220612230430.460 DEBUG storage - storage/storage.c:get_storage_by_name:209 - Detected rootfs type “dir”
lxc-start dut4lan2 20220612230430.461 INFO cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1137 - The container process uses “lxc.payload.dut4lan2” as inner and “lxc.payload.dut4lan2” as limit cgroup
lxc-start dut4lan2 20220612230430.462 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWNS
lxc-start dut4lan2 20220612230430.462 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWPID
lxc-start dut4lan2 20220612230430.462 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWUTS
lxc-start dut4lan2 20220612230430.463 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWIPC
lxc-start dut4lan2 20220612230430.463 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWNET
lxc-start dut4lan2 20220612230430.463 INFO start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWCGROUP
lxc-start dut4lan2 20220612230430.463 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved mnt namespace via fd 19 and stashed path as mnt:/proc/14676/fd/19
lxc-start dut4lan2 20220612230430.463 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved pid namespace via fd 20 and stashed path as pid:/proc/14676/fd/20
lxc-start dut4lan2 20220612230430.463 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved uts namespace via fd 21 and stashed path as uts:/proc/14676/fd/21
lxc-start dut4lan2 20220612230430.463 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved ipc namespace via fd 22 and stashed path as ipc:/proc/14676/fd/22
lxc-start dut4lan2 20220612230430.463 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved net namespace via fd 23 and stashed path as net:/proc/14676/fd/23
lxc-start dut4lan2 20220612230430.463 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved cgroup namespace via fd 24 and stashed path as cgroup:/proc/14676/fd/24
lxc-start dut4lan2 20220612230430.463 WARN cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits_legacy:2767 - Invalid argument - Ignoring legacy cgroup limits on pure cgroup2 system
lxc-start dut4lan2 20220612230430.473 INFO network - network.c:netdev_configure_server_veth:655 - Retrieved mtu 1500 from lxcbr0
lxc-start dut4lan2 20220612230430.474 INFO network - network.c:netdev_configure_server_veth:721 - Attached “vethP58LPO” to bridge “lxcbr0”
lxc-start dut4lan2 20220612230430.474 DEBUG network - network.c:netdev_configure_server_veth:852 - Instantiated veth tunnel “vethP58LPO <–> vethSAURd8”
lxc-start dut4lan2 20220612230430.475 DEBUG conf - conf.c:lxc_mount_rootfs:1436 - Mounted rootfs “/home/umtts/comcast/tsport/dut4lan2/rootfs” onto “/usr/lib/x86_64-linux-gnu/lxc” with options “(null)”
lxc-start dut4lan2 20220612230430.475 INFO conf - conf.c:setup_utsname:875 - Set hostname to “dut4lan2”
lxc-start dut4lan2 20220612230430.504 DEBUG network - network.c:setup_hw_addr:3821 - Mac address “00:16:3e:bb:f0:d5” on “eth0” has been setup
lxc-start dut4lan2 20220612230430.504 DEBUG network - network.c:lxc_network_setup_in_child_namespaces_common:3962 - Network device “eth0” has been setup
lxc-start dut4lan2 20220612230430.504 INFO network - network.c:lxc_setup_network_in_child_namespaces:4019 - Finished setting up network devices with caller assigned names
lxc-start dut4lan2 20220612230430.504 INFO conf - conf.c:mount_autodev:1219 - Preparing “/dev”
lxc-start dut4lan2 20220612230430.504 INFO conf - conf.c:mount_autodev:1280 - Prepared “/dev”
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:mount_entry:2479 - Mounted “proc” on “/usr/lib/x86_64-linux-gnu/lxc/proc” with filesystem type “proc”
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:mount_entry:2479 - Mounted “sysfs” on “/usr/lib/x86_64-linux-gnu/lxc/sys” with filesystem type “sysfs”
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:mount_entry:2416 - Remounting “/sys/fs/fuse/connections” on “/usr/lib/x86_64-linux-gnu/lxc/sys/fs/fuse/connections” to respect bind or remount options
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:mount_entry:2435 - Flags for “/sys/fs/fuse/connections” were 4110, required extra flags are 14
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:mount_entry:2479 - Mounted “/sys/fs/fuse/connections” on “/usr/lib/x86_64-linux-gnu/lxc/sys/fs/fuse/connections” with filesystem type “none”
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:mount_entry:2416 - Remounting “/sys/kernel/debug” on “/usr/lib/x86_64-linux-gnu/lxc/sys/kernel/debug” to respect bind or remount options
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:mount_entry:2435 - Flags for “/sys/kernel/debug” were 4110, required extra flags are 14
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:mount_entry:2479 - Mounted “/sys/kernel/debug” on “/usr/lib/x86_64-linux-gnu/lxc/sys/kernel/debug” with filesystem type “none”
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:mount_entry:2416 - Remounting “/sys/kernel/security” on “/usr/lib/x86_64-linux-gnu/lxc/sys/kernel/security” to respect bind or remount options
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:mount_entry:2435 - Flags for “/sys/kernel/security” were 4110, required extra flags are 14
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:mount_entry:2479 - Mounted “/sys/kernel/security” on “/usr/lib/x86_64-linux-gnu/lxc/sys/kernel/security” with filesystem type “none”
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:mount_entry:2416 - Remounting “/sys/fs/pstore” on “/usr/lib/x86_64-linux-gnu/lxc/sys/fs/pstore” to respect bind or remount options
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:mount_entry:2435 - Flags for “/sys/fs/pstore” were 4110, required extra flags are 14
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:mount_entry:2479 - Mounted “/sys/fs/pstore” on “/usr/lib/x86_64-linux-gnu/lxc/sys/fs/pstore” with filesystem type “none”
lxc-start dut4lan2 20220612230430.505 INFO conf - conf.c:lxc_fill_autodev:1317 - Populating “/dev”
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:lxc_fill_autodev:1326 - Created device node “full”
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:lxc_fill_autodev:1326 - Created device node “null”
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:lxc_fill_autodev:1326 - Created device node “random”
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:lxc_fill_autodev:1326 - Created device node “tty”
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:lxc_fill_autodev:1326 - Created device node “urandom”
lxc-start dut4lan2 20220612230430.505 DEBUG conf - conf.c:lxc_fill_autodev:1326 - Created device node “zero”
lxc-start dut4lan2 20220612230430.505 INFO conf - conf.c:lxc_fill_autodev:1405 - Populated “/dev”
lxc-start dut4lan2 20220612230430.505 INFO conf - conf.c:lxc_transient_proc:3775 - Caller’s PID is 1; /proc/self points to 1
lxc-start dut4lan2 20220612230430.506 DEBUG conf - conf.c:lxc_setup_devpts_child:1751 - Attached detached devpts mount 22 to 20/pts
lxc-start dut4lan2 20220612230430.506 DEBUG conf - conf.c:lxc_setup_devpts_child:1837 - Created “/dev/ptmx” file as bind mount target
lxc-start dut4lan2 20220612230430.506 DEBUG conf - conf.c:lxc_setup_devpts_child:1844 - Bind mounted “/dev/pts/ptmx” to “/dev/ptmx”
lxc-start dut4lan2 20220612230430.506 DEBUG conf - conf.c:lxc_allocate_ttys:1104 - Created tty with ptx fd 24 and pty fd 25 and index 1
lxc-start dut4lan2 20220612230430.506 DEBUG conf - conf.c:lxc_allocate_ttys:1104 - Created tty with ptx fd 26 and pty fd 27 and index 2
lxc-start dut4lan2 20220612230430.506 DEBUG conf - conf.c:lxc_allocate_ttys:1104 - Created tty with ptx fd 28 and pty fd 29 and index 3
lxc-start dut4lan2 20220612230430.506 DEBUG conf - conf.c:lxc_allocate_ttys:1104 - Created tty with ptx fd 30 and pty fd 31 and index 4
lxc-start dut4lan2 20220612230430.506 INFO conf - conf.c:lxc_allocate_ttys:1109 - Finished creating 4 tty devices
lxc-start dut4lan2 20220612230430.506 DEBUG conf - conf.c:lxc_setup_ttys:1028 - Bind mounted “pts/1” onto “/dev/lxc/tty1”
lxc-start dut4lan2 20220612230430.506 DEBUG conf - conf.c:lxc_setup_ttys:1028 - Bind mounted “pts/2” onto “/dev/lxc/tty2”
lxc-start dut4lan2 20220612230430.506 DEBUG conf - conf.c:lxc_setup_ttys:1028 - Bind mounted “pts/3” onto “/dev/lxc/tty3”
lxc-start dut4lan2 20220612230430.506 DEBUG conf - conf.c:lxc_setup_ttys:1028 - Bind mounted “pts/4” onto “/dev/lxc/tty4”
lxc-start dut4lan2 20220612230430.506 INFO conf - conf.c:lxc_setup_ttys:1072 - Finished setting up 4 /dev/tty device(s)
lxc-start dut4lan2 20220612230430.507 INFO conf - conf.c:setup_personality:1917 - Set personality to “0lx0”
lxc-start dut4lan2 20220612230430.507 DEBUG conf - conf.c:capabilities_deny:3200 - Dropped sys_module (16) capability
lxc-start dut4lan2 20220612230430.507 DEBUG conf - conf.c:capabilities_deny:3200 - Dropped mac_admin (33) capability
lxc-start dut4lan2 20220612230430.507 DEBUG conf - conf.c:capabilities_deny:3200 - Dropped mac_override (32) capability
lxc-start dut4lan2 20220612230430.507 DEBUG conf - conf.c:capabilities_deny:3200 - Dropped sys_time (25) capability
lxc-start dut4lan2 20220612230430.507 DEBUG conf - conf.c:capabilities_deny:3203 - Capabilities have been setup
lxc-start dut4lan2 20220612230430.507 NOTICE conf - conf.c:lxc_setup:4469 - The container “dut4lan2” is set up
lxc-start dut4lan2 20220612230430.507 INFO apparmor - lsm/apparmor.c:apparmor_process_label_set_at:1186 - Set AppArmor label to “unconfined”
lxc-start dut4lan2 20220612230430.507 INFO apparmor - lsm/apparmor.c:apparmor_process_label_set:1231 - Changed AppArmor profile to unconfined
lxc-start dut4lan2 20220612230430.507 WARN cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits_legacy:2767 - Invalid argument - Ignoring legacy cgroup limits on pure cgroup2 system
lxc-start dut4lan2 20220612230430.507 DEBUG terminal - terminal.c:lxc_terminal_peer_default:695 - No such device - The process does not have a controlling terminal
lxc-start dut4lan2 20220612230430.508 NOTICE utils - utils.c:lxc_drop_groups:1368 - Dropped supplimentary groups
lxc-start dut4lan2 20220612230430.508 NOTICE start - start.c:start:2161 - Exec’ing “/sbin/init”
lxc-start dut4lan2 20220612230430.508 NOTICE start - start.c:post_start:2172 - Started “/sbin/init” with pid “14677”
lxc-start dut4lan2 20220612230430.516 DEBUG start - start.c:signal_handler:467 - Container init process 14677 exited
lxc-start dut4lan2 20220612230430.516 INFO error - error.c:lxc_error_set_and_log:31 - Child <14677> ended on error (255)
lxc-start dut4lan2 20220612230430.587 INFO network - network.c:lxc_delete_network_priv:3680 - Removed interface “vethP58LPO” from “lxcbr0”
lxc-start dut4lan2 20220612230430.587 DEBUG network - network.c:lxc_delete_network:4173 - Deleted network devices

I’m not seeing an actual error above. How exactly does it fail?

I used the following command to check the log.

sudo lxc-start -n dut4lan2 -P /home/umtts/comcast/tsport/ -l DEBUG -o log

what i noticed that the last 4 lines were diffrent in failed lxc-start.

lxc-start dut4lan2 20220612230430.516 DEBUG start - start.c:signal_handler:467 - Container init process 14677 exited
lxc-start dut4lan2 20220612230430.516 INFO error - error.c:lxc_error_set_and_log:31 - Child <14677> ended on error (255)
lxc-start dut4lan2 20220612230430.587 INFO network - network.c:lxc_delete_network_priv:3680 - Removed interface “vethP58LPO” from “lxcbr0”
lxc-start dut4lan2 20220612230430.587 DEBUG network - network.c:lxc_delete_network:4173 - Deleted network devices

Please let me know if there are methods to get more logs for further debuging

Thanks

This is still not showing an actual error. It just says that your container’s init system has exited.

You may want to look at log files inside of the container’s rootfs or something as LXC is performing perfectly fine above, it’s the container’s init system (systemd in this case) which is deciding to exit causing the container to stop.

My guess is that it’s some kind of system resources being depleted. You can find a list of system knobs that we recommend for production uses here: Production setup - LXD documentation

Thanks for reply Stephen.
I changed all the parameters suggested in production-setup and rebooted the server but i could not bring up all the LXCs.

i see there are many logs in rootfd/var/log folder some are in binary , which log may provide the clue ?
The server is 8 core , 32GiG RAM , top result shows that lot of free RAM. no processes are running inside containers. on the same sys config i have 32 containers runnning on lxc1.0.9 /ubuntu14.
I am using the follwoing config.
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = lxcbr0
lxc.net.0.hwaddr = 00:16:3e:22:34:ec
lxc.net.0.name = eth0
lxc.net.0.ipv4.address = 10.0.3.101/24
lxc.net.0.ipv4.gateway = 10.0.3.1
lxc.mount.entry = proc proc proc nodev,noexec,nosuid 0 0
lxc.mount.entry = sysfs sys sysfs defaults 0 0
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none bind,optional 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none bind,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none bind,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none bind,optional 0 0
lxc.tty.max = 4
lxc.pty.max = 1024
lxc.tty.dir = lxc
lxc.arch = x86_64
lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = c : m
lxc.cgroup.devices.allow = b : m
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 254:0 rm
lxc.cgroup.devices.allow = c 10:229 rwm
lxc.cgroup.devices.allow = c 10:200 rwm
lxc.cgroup.devices.allow = c 1:7 rwm
lxc.cgroup.devices.allow = c 10:228 rwm
lxc.cgroup.devices.allow = c 10:232 rwm
lxc.cap.drop = sys_module
lxc.cap.drop = mac_admin
lxc.cap.drop = mac_override
lxc.cap.drop = sys_time
lxc.apparmor.profile = unconfined
lxc.rootfs.path = dir:/home//tsport/dut1lan1/rootfs
lxc.uts.name = dut1lan1
#lxc.mount.fstab = /home//tsport/dut1lan1/fstab

Please advise.

Thanks

Hi Stephen,
I took out all the configuration related to cgroup

lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = c : m
lxc.cgroup.devices.allow = b : m
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 254:0 rm
lxc.cgroup.devices.allow = c 10:229 rwm
lxc.cgroup.devices.allow = c 10:200 rwm
lxc.cgroup.devices.allow = c 1:7 rwm
lxc.cgroup.devices.allow = c 10:228 rwm
lxc.cgroup.devices.allow = c 10:232 rwm

and i could bring up all now , iam afraid i may break something ,
what may be the issue ?

Thanks

Hello i’ve found this limit even by taking out lxc.cgroup.devices.allow limit still there ;
My OS ubuntu 22.04 upgraded from 20.04 with cgroup2 disable to start debian 11 lxc.
without cgroup2 disabled debian 11 don’t start.
any help would be appreciated.

By upgrading from 5.15.0-37-generic to 5.15.0-39-generic
issue is solved