Ulimit nofiles in centos 7.5 containers a systemd bug?

Been searching the forums and google for a workaround as centos 7.5 lxd container’s processes all seem to have open files nofile limit of 65536. However, ubuntu 18.04 lxd container’s processes inherit the ubuntu 18.04 host’s nofile max limit of 1048576. Is this related to centos 7.5’s older systemd version for https://github.com/systemd/systemd/pull/5795 ? Best workaround for centos 7.5 lxd guest containers ?

centos 7.5 guest systemd version

lxc exec centos75 -- systemctl --version
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN

ubuntu 18.04 guest systemd version

lxc exec ubuntu18 -- systemctl --version        
systemd 237
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid

looks like fix for systemd was put in 229 https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1686361 ?

Not sure about workaround, but the fix was indeed done directly in systemd.
Might be worth trying to get the systemd maintainer for CentOS to include that particular fix.

1 Like

cheers… found a solution thanks to facebook’s shared centos 7 rpm backports for updating systemd to 234 https://maciej.lasyk.info/2016/Dec/16/systemd-231-latest-in-centos-7-thx-to-facebook/

seems like lxd container is perfect place to test something like this out :slight_smile:

wget https://copr.fedorainfracloud.org/coprs/jsynacek/systemd-backports-for-centos-7/repo/epel-7/jsynacek-systemd-backports-for-centos-7-epel-7.repo -O /etc/yum.repos.d/jsynacek-systemd-centos-7.repo

yum -y update systemd
lxc exec centos75 -- systemctl --version                         
systemd 234
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN default-hierarchy=hybrid

restarting centos75 container now shows my LEMP stack scripts increased nofile 524288 for nginx server

lxc restart centos75

# check limits for nginx pid
prlimit -p 1553
RESOURCE   DESCRIPTION                             SOFT      HARD UNITS
AS         address space limit                unlimited unlimited bytes
CORE       max core file size                         0 unlimited bytes
CPU        CPU time                           unlimited unlimited seconds
DATA       max data size                      unlimited unlimited bytes
FSIZE      max file size                      unlimited unlimited bytes
LOCKS      max number of file locks held      unlimited unlimited locks
MEMLOCK    max locked-in-memory address space  16777216  16777216 bytes
MSGQUEUE   max bytes in POSIX mqueues            819200    819200 bytes
NICE       max nice prio allowed to raise             0         0 
NOFILE     max number of open files              524288    524288 files
NPROC      max number of processes            unlimited unlimited processes
RSS        max resident set size              unlimited unlimited bytes
RTPRIO     max real-time priority                     0         0 
RTTIME     timeout for real-time tasks        unlimited unlimited microsecs
SIGPENDING max number of pending signals          63928     63928 signals
STACK      max stack size                       8388608 unlimited bytes

CentOS 7.5 container Nginx server

nginx -V
nginx version: nginx/1.13.12 (050618-001557)
built by gcc 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC)
built with OpenSSL 1.1.0h 27 Mar 2018
TLS SNI support enabled
configure arguments: --with-ld-opt=’-ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib’ --with-cc-opt=’-m64 -march=native -DTCP_FASTOPEN=23 -g -O3 -fstack-protector-strong -flto -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wimplicit-fallthrough=0 -fcode-hoisting -Wp,-D_FORTIFY_SOURCE=2 -gsplit-dwarf’ --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --build=050618-001557 --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-libatomic --with-http_gzip_static_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream=dynamic --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=…/ngx-fancyindex-0.4.2 --add-module=…/ngx_cache_purge-2.4.2 --add-module=…/ngx_devel_kit-0.3.0 --add-dynamic-module=…/set-misc-nginx-module-0.32 --add-dynamic-module=…/echo-nginx-module-0.61 --add-module=…/redis2-nginx-module-0.15 --add-module=…/ngx_http_redis-0.3.7 --add-module=…/memc-nginx-module-0.18 --add-module=…/srcache-nginx-module-0.31 --add-dynamic-module=…/headers-more-nginx-module-0.33 --with-pcre=…/pcre-8.42 --with-pcre-jit --with-zlib=…/zlib-cloudflare-1.3.0 --with-http_ssl_module --with-http_v2_module --with-openssl=…/openssl-1.1.0h --with-openssl-opt=‘enable-ec_nistp_64_gcc_128’

so for centos 7.5 containers my work flow would be to create a golden base image with workarounds and use that to launch centos 7.5 containers

create centos75-base container to add all workarounds then create/publish image centos7-systemdfix

# create a golden base centos 75 lxd container image to work off of
lxc profile set default security.syscalls.blacklist "keyctl errno 38"
lxc launch images:centos/7 centos75-base
lxc exec centos75-base -- echo "export LANG=en_US.UTF-8" >> /etc/profile.d/locale.sh
lxc exec centos75-base -- echo "export LANGUAGE=en_US.UTF-8" >> /etc/profile.d/locale.sh
lxc exec centos75-base -- source /etc/profile.d/locale.sh
lxc exec centos75-base -- sed -i "s|plugins=1|plugins=1\nexclude=\*.i386 \*.i586 \*.i686|" /etc/yum.conf
lxc exec centos75-base -- yum -y update
lxc exec centos75-base -- yum -y install wget openssh openssh-server curl curl-devel libcurl libcurl-devel
lxc exec centos75-base -- wget https://copr.fedorainfracloud.org/coprs/jsynacek/systemd-backports-for-centos-7/repo/epel-7/jsynacek-systemd-backports-for-centos-7-epel-7.repo -O /etc/yum.repos.d/jsynacek-systemd-centos-7.repo
lxc exec centos75-base -- yum -y update systemd
lxc exec centos75-base -- yum -y install openssh openssh-server
lxc exec centos75-base -- systemctl enable sshd
lxc exec centos75-base -- systemctl restart sshd
lxc exec centos75-base -- systemctl status sshd
lxc restart centos75-base
lxc publish centos75-base --alias centos7-systemdfix --force
lxc delete centos75-base --force

use centos7-systemdfix image to create container centos75

lxc launch centos7-systemdfix centos75
lxc config set centos75 boot.autostart true

lxc exec centos75 -- systemctl --version
systemd 234
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN default-hierarchy=hybrid

edit: may have some issues with updated systemd 234 though Centos 7.5 container operation not permitted?