Can not bring up more than 11 LXC containers on ubuntu22

pm16 · June 12, 2022, 11:16pm

I have created 30+ copies of a LXC container but for some reason i can not bring up more that 11 LXCs , is there any limitation i shoud be aware of ? Please see the attached log.
I can bring up any of the 11 LXCs from 30 created but the 12th one fail to come up.

Log

lxc-start dut4lan2 20220612230430.440 INFO lxc-start dut4lan2 20220612230430.440 DEBUG lxc-start dut4lan2 20220612230430.441 INFO lxc-start dut4lan2 20220612230430.441 INFO lxc-start dut4lan2 20220612230430.460 INFO lxc-start dut4lan2 20220612230430.460 DEBUG lxc-start dut4lan2 20220612230430.461 INFO lxc-start dut4lan2 20220612230430.462 INFO lxc-start dut4lan2 20220612230430.462 INFO lxc-start dut4lan2 20220612230430.462 INFO lxc-start dut4lan2 20220612230430.463 INFO lxc-start dut4lan2 20220612230430.463 INFO lxc-start dut4lan2 20220612230430.463 INFO lxc-start dut4lan2 20220612230430.463 DEBUG lxc-start dut4lan2 20220612230430.463 DEBUG lxc-start dut4lan2 20220612230430.463 DEBUG lxc-start dut4lan2 20220612230430.463 DEBUG lxc-start dut4lan2 20220612230430.463 DEBUG lxc-start dut4lan2 20220612230430.463 DEBUG lxc-start dut4lan2 20220612230430.463 WARN lxc-start dut4lan2 20220612230430.473 INFO lxc-start dut4lan2 20220612230430.474 INFO lxc-start dut4lan2 20220612230430.474 DEBUG lxc-start dut4lan2 20220612230430.475 DEBUG lxc-start dut4lan2 20220612230430.475 INFO lxc-start dut4lan2 20220612230430.504 DEBUG lxc-start dut4lan2 20220612230430.504 DEBUG lxc-start dut4lan2 20220612230430.504 INFO lxc-start dut4lan2 20220612230430.504 INFO lxc-start dut4lan2 20220612230430.504 INFO lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 INFO lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 DEBUG lxc-start dut4lan2 20220612230430.505 INFO lxc-start dut4lan2 20220612230430.505 INFO lxc-start dut4lan2 20220612230430.506 DEBUG lxc-start dut4lan2 20220612230430.506 DEBUG lxc-start dut4lan2 20220612230430.506 DEBUG lxc-start dut4lan2 20220612230430.506 DEBUG lxc-start dut4lan2 20220612230430.506 DEBUG lxc-start dut4lan2 20220612230430.506 DEBUG lxc-start dut4lan2 20220612230430.506 DEBUG lxc-start dut4lan2 20220612230430.506 INFO lxc-start dut4lan2 20220612230430.506 DEBUG lxc-start dut4lan2 20220612230430.506 DEBUG lxc-start dut4lan2 20220612230430.506 DEBUG lxc-start dut4lan2 20220612230430.506 DEBUG lxc-start dut4lan2 20220612230430.506 INFO lxc-start dut4lan2 20220612230430.507 INFO lxc-start dut4lan2 20220612230430.507 DEBUG lxc-start dut4lan2 20220612230430.507 DEBUG lxc-start dut4lan2 20220612230430.507 DEBUG lxc-start dut4lan2 20220612230430.507 DEBUG lxc-start dut4lan2 20220612230430.507 DEBUG lxc-start dut4lan2 20220612230430.507 NOTICE lxc-start dut4lan2 20220612230430.507 INFO lxc-start dut4lan2 20220612230430.507 INFO lxc-start dut4lan2 20220612230430.507 WARN lxc-start dut4lan2 20220612230430.507 DEBUG lxc-start dut4lan2 20220612230430.508 NOTICE lxc-start dut4lan2 20220612230430.508 NOTICE lxc-start dut4lan2 20220612230430.508 NOTICE lxc-start dut4lan2 20220612230430.516 DEBUG lxc-start dut4lan2 20220612230430.516 INFO lxc-start dut4lan2 20220612230430.587 INFO lxc-start dut4lan2 20220612230430.587 DEBUG lxccontainer - lxccontainer.c:do_lxcapi_start:997 - Set process title to [lxc monitor] /home/umtts/comcast/tsport dut4lan2
lxccontainer - lxccontainer.c:wait_on_daemonized_start:858 - First child 14675 exited
lsm - lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
start - start.c:lxc_init:884 - Container “dut4lan2” is initialized
cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1029 - The monitor process uses “lxc.monitor.dut4lan2” as cgroup
storage - storage/storage.c:get_storage_by_name:209 - Detected rootfs type “dir”
cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1137 - The container process uses “lxc.payload.dut4lan2” as inner and “lxc.payload.dut4lan2” as limit cgroup
start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWNS
start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWPID
start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWUTS
start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWIPC
start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWNET
start - start.c:lxc_spawn:1765 - Cloned CLONE_NEWCGROUP
start - start.c:lxc_try_preserve_namespace:139 - Preserved mnt namespace via fd 19 and stashed path as mnt:/proc/14676/fd/19
start - start.c:lxc_try_preserve_namespace:139 - Preserved pid namespace via fd 20 and stashed path as pid:/proc/14676/fd/20
start - start.c:lxc_try_preserve_namespace:139 - Preserved uts namespace via fd 21 and stashed path as uts:/proc/14676/fd/21
start - start.c:lxc_try_preserve_namespace:139 - Preserved ipc namespace via fd 22 and stashed path as ipc:/proc/14676/fd/22
start - start.c:lxc_try_preserve_namespace:139 - Preserved net namespace via fd 23 and stashed path as net:/proc/14676/fd/23
start - start.c:lxc_try_preserve_namespace:139 - Preserved cgroup namespace via fd 24 and stashed path as cgroup:/proc/14676/fd/24
cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits_legacy:2767 - Invalid argument - Ignoring legacy cgroup limits on pure cgroup2 system
network - network.c:netdev_configure_server_veth:655 - Retrieved mtu 1500 from lxcbr0
network - network.c:netdev_configure_server_veth:721 - Attached “vethP58LPO” to bridge “lxcbr0”
network - network.c:netdev_configure_server_veth:852 - Instantiated veth tunnel “vethP58LPO <–> vethSAURd8”
conf - conf.c:lxc_mount_rootfs:1436 - Mounted rootfs “/home/umtts/comcast/tsport/dut4lan2/rootfs” onto “/usr/lib/x86_64-linux-gnu/lxc” with options “(null)”
conf - conf.c:setup_utsname:875 - Set hostname to “dut4lan2”
network - network.c:setup_hw_addr:3821 - Mac address “00:16:3e:bb:f0:d5” on “eth0” has been setup
network - network.c:lxc_network_setup_in_child_namespaces_common:3962 - Network device “eth0” has been setup
network - network.c:lxc_setup_network_in_child_namespaces:4019 - Finished setting up network devices with caller assigned names
conf - conf.c:mount_autodev:1219 - Preparing “/dev”
conf - conf.c:mount_autodev:1280 - Prepared “/dev”
conf - conf.c:mount_entry:2479 - Mounted “proc” on “/usr/lib/x86_64-linux-gnu/lxc/proc” with filesystem type “proc”
conf - conf.c:mount_entry:2479 - Mounted “sysfs” on “/usr/lib/x86_64-linux-gnu/lxc/sys” with filesystem type “sysfs”
conf - conf.c:mount_entry:2416 - Remounting “/sys/fs/fuse/connections” on “/usr/lib/x86_64-linux-gnu/lxc/sys/fs/fuse/connections” to respect bind or remount options
conf - conf.c:mount_entry:2435 - Flags for “/sys/fs/fuse/connections” were 4110, required extra flags are 14
conf - conf.c:mount_entry:2479 - Mounted “/sys/fs/fuse/connections” on “/usr/lib/x86_64-linux-gnu/lxc/sys/fs/fuse/connections” with filesystem type “none”
conf - conf.c:mount_entry:2416 - Remounting “/sys/kernel/debug” on “/usr/lib/x86_64-linux-gnu/lxc/sys/kernel/debug” to respect bind or remount options
conf - conf.c:mount_entry:2435 - Flags for “/sys/kernel/debug” were 4110, required extra flags are 14
conf - conf.c:mount_entry:2479 - Mounted “/sys/kernel/debug” on “/usr/lib/x86_64-linux-gnu/lxc/sys/kernel/debug” with filesystem type “none”
conf - conf.c:mount_entry:2416 - Remounting “/sys/kernel/security” on “/usr/lib/x86_64-linux-gnu/lxc/sys/kernel/security” to respect bind or remount options
conf - conf.c:mount_entry:2435 - Flags for “/sys/kernel/security” were 4110, required extra flags are 14
conf - conf.c:mount_entry:2479 - Mounted “/sys/kernel/security” on “/usr/lib/x86_64-linux-gnu/lxc/sys/kernel/security” with filesystem type “none”
conf - conf.c:mount_entry:2416 - Remounting “/sys/fs/pstore” on “/usr/lib/x86_64-linux-gnu/lxc/sys/fs/pstore” to respect bind or remount options
conf - conf.c:mount_entry:2435 - Flags for “/sys/fs/pstore” were 4110, required extra flags are 14
conf - conf.c:mount_entry:2479 - Mounted “/sys/fs/pstore” on “/usr/lib/x86_64-linux-gnu/lxc/sys/fs/pstore” with filesystem type “none”
conf - conf.c:lxc_fill_autodev:1317 - Populating “/dev”
conf - conf.c:lxc_fill_autodev:1326 - Created device node “full”
conf - conf.c:lxc_fill_autodev:1326 - Created device node “null”
conf - conf.c:lxc_fill_autodev:1326 - Created device node “random”
conf - conf.c:lxc_fill_autodev:1326 - Created device node “tty”
conf - conf.c:lxc_fill_autodev:1326 - Created device node “urandom”
conf - conf.c:lxc_fill_autodev:1326 - Created device node “zero”
conf - conf.c:lxc_fill_autodev:1405 - Populated “/dev”
conf - conf.c:lxc_transient_proc:3775 - Caller’s PID is 1; /proc/self points to 1
conf - conf.c:lxc_setup_devpts_child:1751 - Attached detached devpts mount 22 to 20/pts
conf - conf.c:lxc_setup_devpts_child:1837 - Created “/dev/ptmx” file as bind mount target
conf - conf.c:lxc_setup_devpts_child:1844 - Bind mounted “/dev/pts/ptmx” to “/dev/ptmx”
conf - conf.c:lxc_allocate_ttys:1104 - Created tty with ptx fd 24 and pty fd 25 and index 1
conf - conf.c:lxc_allocate_ttys:1104 - Created tty with ptx fd 26 and pty fd 27 and index 2
conf - conf.c:lxc_allocate_ttys:1104 - Created tty with ptx fd 28 and pty fd 29 and index 3
conf - conf.c:lxc_allocate_ttys:1104 - Created tty with ptx fd 30 and pty fd 31 and index 4
conf - conf.c:lxc_allocate_ttys:1109 - Finished creating 4 tty devices
conf - conf.c:lxc_setup_ttys:1028 - Bind mounted “pts/1” onto “/dev/lxc/tty1”
conf - conf.c:lxc_setup_ttys:1028 - Bind mounted “pts/2” onto “/dev/lxc/tty2”
conf - conf.c:lxc_setup_ttys:1028 - Bind mounted “pts/3” onto “/dev/lxc/tty3”
conf - conf.c:lxc_setup_ttys:1028 - Bind mounted “pts/4” onto “/dev/lxc/tty4”
conf - conf.c:lxc_setup_ttys:1072 - Finished setting up 4 /dev/tty device(s)
conf - conf.c:setup_personality:1917 - Set personality to “0lx0”
conf - conf.c:capabilities_deny:3200 - Dropped sys_module (16) capability
conf - conf.c:capabilities_deny:3200 - Dropped mac_admin (33) capability
conf - conf.c:capabilities_deny:3200 - Dropped mac_override (32) capability
conf - conf.c:capabilities_deny:3200 - Dropped sys_time (25) capability
conf - conf.c:capabilities_deny:3203 - Capabilities have been setup
conf - conf.c:lxc_setup:4469 - The container “dut4lan2” is set up
apparmor - lsm/apparmor.c:apparmor_process_label_set_at:1186 - Set AppArmor label to “unconfined”
apparmor - lsm/apparmor.c:apparmor_process_label_set:1231 - Changed AppArmor profile to unconfined
cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits_legacy:2767 - Invalid argument - Ignoring legacy cgroup limits on pure cgroup2 system
terminal - terminal.c:lxc_terminal_peer_default:695 - No such device - The process does not have a controlling terminal
utils - utils.c:lxc_drop_groups:1368 - Dropped supplimentary groups
start - start.c:start:2161 - Exec’ing “/sbin/init”
start - start.c:post_start:2172 - Started “/sbin/init” with pid “14677”
start - start.c:signal_handler:467 - Container init process 14677 exited
error - error.c:lxc_error_set_and_log:31 - Child <14677> ended on error (255)
network - network.c:lxc_delete_network_priv:3680 - Removed interface “vethP58LPO” from “lxcbr0”
network - network.c:lxc_delete_network:4173 - Deleted network devices

stgraber · June 13, 2022, 2:03am

I’m not seeing an actual error above. How exactly does it fail?

pm16 · June 13, 2022, 7:00am

I used the following command to check the log.

sudo lxc-start -n dut4lan2 -P /home/umtts/comcast/tsport/ -l DEBUG -o log

what i noticed that the last 4 lines were diffrent in failed lxc-start.

lxc-start dut4lan2 20220612230430.516 DEBUG start - start.c:signal_handler:467 - Container init process 14677 exited
lxc-start dut4lan2 20220612230430.516 INFO error - error.c:lxc_error_set_and_log:31 - Child <14677> ended on error (255)
lxc-start dut4lan2 20220612230430.587 INFO network - network.c:lxc_delete_network_priv:3680 - Removed interface “vethP58LPO” from “lxcbr0”
lxc-start dut4lan2 20220612230430.587 DEBUG network - network.c:lxc_delete_network:4173 - Deleted network devices

Please let me know if there are methods to get more logs for further debuging

Thanks

stgraber · June 13, 2022, 4:41pm

This is still not showing an actual error. It just says that your container’s init system has exited.

You may want to look at log files inside of the container’s rootfs or something as LXC is performing perfectly fine above, it’s the container’s init system (systemd in this case) which is deciding to exit causing the container to stop.

My guess is that it’s some kind of system resources being depleted. You can find a list of system knobs that we recommend for production uses here: Production setup - LXD documentation

pm16 · June 14, 2022, 3:46am

Thanks for reply Stephen.
I changed all the parameters suggested in production-setup and rebooted the server but i could not bring up all the LXCs.

i see there are many logs in rootfd/var/log folder some are in binary , which log may provide the clue ?
The server is 8 core , 32GiG RAM , top result shows that lot of free RAM. no processes are running inside containers. on the same sys config i have 32 containers runnning on lxc1.0.9 /ubuntu14.
I am using the follwoing config.
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = lxcbr0
lxc.net.0.hwaddr = 00:16:3e:22:34:ec
lxc.net.0.name = eth0
lxc.net.0.ipv4.address = 10.0.3.101/24
lxc.net.0.ipv4.gateway = 10.0.3.1
lxc.mount.entry = proc proc proc nodev,noexec,nosuid 0 0
lxc.mount.entry = sysfs sys sysfs defaults 0 0
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none bind,optional 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none bind,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none bind,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none bind,optional 0 0
lxc.tty.max = 4
lxc.pty.max = 1024
lxc.tty.dir = lxc
lxc.arch = x86_64
lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = c : m
lxc.cgroup.devices.allow = b : m
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 254:0 rm
lxc.cgroup.devices.allow = c 10:229 rwm
lxc.cgroup.devices.allow = c 10:200 rwm
lxc.cgroup.devices.allow = c 1:7 rwm
lxc.cgroup.devices.allow = c 10:228 rwm
lxc.cgroup.devices.allow = c 10:232 rwm
lxc.cap.drop = sys_module
lxc.cap.drop = mac_admin
lxc.cap.drop = mac_override
lxc.cap.drop = sys_time
lxc.apparmor.profile = unconfined
lxc.rootfs.path = dir:/home//tsport/dut1lan1/rootfs
lxc.uts.name = dut1lan1
#lxc.mount.fstab = /home//tsport/dut1lan1/fstab

Please advise.

Thanks

pm16 · June 15, 2022, 6:59pm

Hi Stephen,
I took out all the configuration related to cgroup

lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = c : m
lxc.cgroup.devices.allow = b : m
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 254:0 rm
lxc.cgroup.devices.allow = c 10:229 rwm
lxc.cgroup.devices.allow = c 10:200 rwm
lxc.cgroup.devices.allow = c 1:7 rwm
lxc.cgroup.devices.allow = c 10:228 rwm
lxc.cgroup.devices.allow = c 10:232 rwm

and i could bring up all now , iam afraid i may break something ,
what may be the issue ?

Thanks

pmarques · June 17, 2022, 1:19pm

Hello i’ve found this limit even by taking out lxc.cgroup.devices.allow limit still there ;
My OS ubuntu 22.04 upgraded from 20.04 with cgroup2 disable to start debian 11 lxc.
without cgroup2 disabled debian 11 don’t start.
any help would be appreciated.

pmarques · June 17, 2022, 1:41pm

By upgrading from 5.15.0-37-generic to 5.15.0-39-generic
issue is solved

pm16 · June 25, 2022, 7:04pm

I think the limit on number of LXCs can be brough up on ubuntu22 host is limited by unknow system resource , if anyone is aware of the system parameter which restrict the LXC on ubuntu22 , please share.
i have a server with 32 gig ram 8 core cpu and i could bring up 15 to 18 LXCs only.