Centos 7.5 container operation not permitted?

LXD
1

Seems my centos 7.5 lxd guest container LEMP install has alot of operation not permitted errors i.e. mariadb service not starting ?

uname -a
Linux centos75 4.15.0-22-generic #24-Ubuntu SMP Wed May 16 12:15:17 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

lxc exec centos75 -- systemctl --version                         
systemd 234
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN default-hierarchy=hybrid

lxc exec centos75 -- systemctl list-units --failed
  UNIT                          LOAD   ACTIVE SUB    DESCRIPTION                                 
â—Ź sys-kernel-config.mount       loaded failed failed Kernel Configuration File System            
â—Ź haveged.service               loaded failed failed Entropy Daemon based on the HAVEGE algorithm
â—Ź mariadb.service               loaded failed failed MariaDB 10.1.33 database server             
â—Ź rngd.service                  loaded failed failed Hardware RNG Entropy Gatherer Daemon        
â—Ź systemd-remount-fs.service    loaded failed failed Remount Root and Kernel File Systems        
â—Ź user@0.service                loaded failed failed User Manager for UID 0                      
â—Ź systemd-journald-audit.socket loaded failed failed Journal Audit Socket                        

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

7 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

Jun 05 08:16:39 centos75 systemd[1]: mariadb.service: Failed to reset devices.list: Operation not permitted
Jun 05 08:16:39 centos75 systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jun 05 08:16:39 centos75 systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jun 05 08:16:39 centos75 systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jun 05 08:16:39 centos75 systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jun 05 08:16:39 centos75 systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jun 05 08:16:39 centos75 systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jun 05 08:16:39 centos75 systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jun 05 08:16:39 centos75 systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jun 05 08:16:39 centos75 systemd[1]: mariadb.service: Failed to set invocation ID on control group /system.slice/mariadb.service, ignoring: Operation not permitted
Jun 05 08:16:39 centos75 systemd[1]: Starting MariaDB 10.1.33 database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mariadb.service has begun starting up.
Jun 05 08:16:39 centos75 systemd[1]: run-user-0.mount: Failed to reset devices.list: Operation not permitted
Jun 05 08:16:39 centos75 systemd[1]: run-user-0.mount: Failed to set invocation ID on control group /system.slice/run-user-0.mount, ignoring: Operation not permitted
Jun 05 08:16:39 centos75 systemd[1836]: mariadb.service: Failed at step KEYRING spawning /bin/sh: Permission denied
-- Subject: Process /bin/sh could not be executed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- The process /bin/sh could not be executed and failed.
-- 
-- The error number returned by this process is 13.
Jun 05 08:16:39 centos75 systemd[1]: mariadb.service: Control process exited, code=exited status=237
Jun 05 08:16:39 centos75 systemd[1]: Failed to start MariaDB 10.1.33 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mariadb.service has failed.
-- 
-- The result is failed.
Jun 05 08:16:39 centos75 systemd[1]: mariadb.service: Unit entered failed state.
Jun 05 08:16:39 centos75 systemd[1]: mariadb.service: Failed with result 'exit-code'.

and

journalctl -xe --no-pager | grep 'Operation not permitted' | awk '{print $6,$7,$8,$9,$10,$11,$12,$13,$14,$15}' | sort | uniq
Failed to reset devices.list on /system.slice/console-getty.service: Operation not permitted 
Failed to reset devices.list on /system.slice/crond.service: Operation not permitted 
Failed to reset devices.list on /system.slice/csf.service: Operation not permitted 
Failed to reset devices.list on /system.slice/dbus.service: Operation not permitted 
Failed to reset devices.list on /system.slice/dev-full.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/dev-fuse.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/dev-lxd.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/dev-.lxd\x2dmounts.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/dev-mqueue.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/dev-net-tun.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/dev-null.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/dev-ptmx.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/dev-random.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/dev-tty.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/dev-urandom.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/dev-zero.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/haveged.service: Operation not permitted 
Failed to reset devices.list on /system.slice/ip6tables.service: Operation not permitted 
Failed to reset devices.list on /system.slice/iptables.service: Operation not permitted 
Failed to reset devices.list on /system.slice/lfd.service: Operation not permitted 
Failed to reset devices.list on /system.slice/mariadb.service: Operation not permitted 
Failed to reset devices.list on /system.slice/-.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/network.service: Operation not permitted 
Failed to reset devices.list on /system.slice/nginx.service: Operation not permitted 
Failed to reset devices.list on /system.slice/ntpd.service: Operation not permitted 
Failed to reset devices.list on /system.slice: Operation not permitted 
Failed to reset devices.list on /system.slice/postfix.service: Operation not permitted 
Failed to reset devices.list on /system.slice/proc-cpuinfo.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/proc-diskstats.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/proc-meminfo.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/proc-stat.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/proc-swaps.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/proc-sys-fs-binfmt_misc.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/proc-uptime.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/pure-ftpd.service: Operation not permitted 
Failed to reset devices.list on /system.slice/rhel-domainname.service: Operation not permitted 
Failed to reset devices.list on /system.slice/rhel-readonly.service: Operation not permitted 
Failed to reset devices.list on /system.slice/rsyslog.service: Operation not permitted 
Failed to reset devices.list on /system.slice/run-user-0.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/sshd.service: Operation not permitted 
Failed to reset devices.list on /system.slice/sys-fs-fuse-connections.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/sys-kernel-debug.mount: Operation not permitted 
Failed to reset devices.list on /system.slice/sysstat.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-hwdb-update.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-journal-catalog-update.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-journald.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-journal-flush.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-localed.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-logind.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-random-seed.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-tmpfiles-setup-dev.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-tmpfiles-setup.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-udevd.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-udev-trigger.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-update-done.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-update-utmp.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-user-sessions.service: Operation not permitted 
Failed to reset devices.list on /system.slice/system-getty.slice: Operation not permitted 
Failed to reset devices.list on /system.slice/tmp.mount: Operation not permitted 
Failed to reset devices.list on /user.slice: Operation not permitted 
Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted 
Failed to set devices.allow on /system.slice/systemd-localed.service: Operation not permitted 
pam_limits(crond:session): Could not set limit for 'nofile': Operation not permitted
2

seems if i downgrade systemd 234 to systemd 219 centos 7.5 default mariadb mysql restarts

edit: seems bug in systemd 234 https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1691096 ?

mv /etc/yum/protected.d/systemd.conf /etc/yum/protected.d/systemd.conf.bak

yum history list   
Loaded plugins: fastestmirror, priorities, versionlock
ID     | Command line             | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
    32 | update systemd --disable | 2018-06-05 05:05 | I, O, U        |   17 EE
    31 | -y -q install haveged rn | 2018-06-05 00:25 | Install        |    3   
    30 | -q -y install pure-ftpd  | 2018-06-05 00:24 | Install        |    3   
    29 | -y install ImageMagick6  | 2018-06-05 00:24 | Install        |    9   
    28 | -q -y install libmemcach | 2018-06-05 00:23 | Install        |    2   
    27 | -q -y install postfix-pe | 2018-06-05 00:22 | Install        |    4   
    26 | -q -y install net-snmp n | 2018-06-05 00:19 | Install        |    1   
    25 | -q -y install libtidy li | 2018-06-05 00:19 | Install        |    2   
    24 | -q -y install libicu lib | 2018-06-05 00:19 | Install        |    2   
    23 | -q -y install fio --disa | 2018-06-05 00:18 | Install        |   13   
    22 | -y install mytop         | 2018-06-05 00:18 | Install        |    1   
    21 | -y install net-snmp --di | 2018-06-05 00:18 | Install        |    1   
    20 | -y install postfix --dis | 2018-06-05 00:18 | Install        |    1   
    19 | -q -y install perl-DBD-M | 2018-06-05 00:18 | Install        |    1   
    18 | -y install MariaDB-clien | 2018-06-05 00:17 | Install        |    7 EE
    17 | -y remove mariadb-libs   | 2018-06-05 00:17 | Erase          |    5 EE
    16 | -y -q install devtoolset | 2018-06-05 00:15 | Install        |   14 EE
    15 | -y -q install centos-rel | 2018-06-05 00:15 | Install        |    2  <
    14 | -y -q install mytop ifto | 2018-06-05 00:13 | Install        |    6 > 
    13 | -y -q install yum-plugin | 2018-06-05 00:13 | Install        |    1   
history list

undo

yum history undo 32

mv -f /etc/yum/protected.d/systemd.conf.bak /etc/yum/protected.d/systemd.conf

restart

systemctl restart mariadb

systemctl status mariadb 
? mariadb.service - MariaDB 10.1.33 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           +-migrated-from-my.cnf-settings.conf, openfileslimit.conf, protecthome.conf
   Active: active (running) since Tue 2018-06-05 08:42:13 UTC; 8s ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 2753 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 2494 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
  Process: 2493 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
 Main PID: 2729 (mysqld)
   Status: "Taking your SQL requests now..."
   CGroup: /init.scope/system.slice/mariadb.service
           +-2729 /usr/sbin/mysqld

Jun 05 08:42:12 centos75 systemd[1]: Starting MariaDB 10.1.33 database server...
Jun 05 08:42:13 centos75 systemd[1]: Started MariaDB 10.1.33 database server.

still though

lxc exec centos75 -- systemctl list-units --failed --all
  UNIT                       LOAD   ACTIVE SUB    DESCRIPTION
<E2><97><8F> dev-hugepages.mount        loaded failed failed Huge Pages File System
<E2><97><8F> sys-kernel-config.mount    loaded failed failed Configuration File System
<E2><97><8F> haveged.service            loaded failed failed Entropy Daemon based on the HAVEGE algorithm
<E2><97><8F> rc-local.service           loaded failed failed /etc/rc.d/rc.local Compatibility
<E2><97><8F> rngd.service               loaded failed failed Hardware RNG Entropy Gatherer Daemon
<E2><97><8F> systemd-remount-fs.service loaded failed failed Remount Root and Kernel File Systems
<E2><97><8F> systemd-sysctl.service     loaded failed failed Apply Kernel Variables

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

7 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'

haveged fails to start though

systemctl status haveged 
â—Ź haveged.service - Entropy Daemon based on the HAVEGE algorithm
   Loaded: loaded (/usr/lib/systemd/system/haveged.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/haveged.service.d
           └─haveged.conf
   Active: failed (Result: exit-code) since Tue 2018-06-05 08:58:12 UTC; 2s ago
     Docs: man:haveged(8)
           http://www.issihosts.com/haveged/
  Process: 1405 ExecStart=/usr/sbin/haveged -w 4067 -v 1 --Foreground (code=exited, status=1/FAILURE)
 Main PID: 1405 (code=exited, status=1/FAILURE)

Jun 05 08:58:11 centos75 systemd[1]: Starting Entropy Daemon based on the HAVEGE algorithm...
Jun 05 08:58:12 centos75 haveged[1405]: haveged: ver: 1.9.1; arch: x86; vend: GenuineIntel; build: (gcc 4.8.2 ITV); collect: 128K
Jun 05 08:58:12 centos75 haveged[1405]: haveged: cpu: (L4 VC); data: 32K (L2 L4 V); inst: 32K (L2 L4 V); idx: 21/40; sz: 32709/60538
Jun 05 08:58:12 centos75 haveged[1405]: haveged: tot tests(BA8): A:1/1 B:1/1 continuous tests(B):  last entropy estimate 8.00456
Jun 05 08:58:12 centos75 haveged[1405]: haveged: fills: 0, generated: 0
Jun 05 08:58:12 centos75 haveged[1405]: haveged: Fail:set_watermark()!
Jun 05 08:58:12 centos75 haveged[1405]: haveged starting up
Jun 05 08:58:12 centos75 systemd[1]: haveged.service: main process exited, code=exited, status=1/FAILURE
Jun 05 08:58:12 centos75 systemd[1]: Unit haveged.service entered failed state.
Jun 05 08:58:12 centos75 systemd[1]: haveged.service failed.
3

At least one of the failures above refers to keyring which is a feature that systemd added recently which doesn’t work in containers and that it doesn’t detect properly…

The workaround for that which may unstick some of your units is:

lxc profile set default security.syscalls.blacklist "keyctl errno 38"

This effectively has the kernel pretend that the syscall systemd is doing simply doesn’t exist.

1 Like
4

cheers after i run that command, do i have to anything for existing lxd containers that are running ?

5

Yeah, you’ll need to restart the container for the new Seccomp policy to apply as we unfortunately can’t easily change those at runtime.

1 Like
6

thanks

also updated to snap lxd 3.1

lxd --version
3.1

just checking with systemd 219 native centos 7.5 version first

lxc profile set default security.syscalls.blacklist "keyctl errno 38"
lxc restart centos75
lxc exec centos75 -- systemctl list-units --failed
  UNIT                       LOAD   ACTIVE SUB    DESCRIPTION
<E2><97><8F> dev-hugepages.mount        loaded failed failed Huge Pages File System
<E2><97><8F> sys-kernel-config.mount    loaded failed failed Configuration File System
<E2><97><8F> haveged.service            loaded failed failed Entropy Daemon based on the HAVEGE algorithm
<E2><97><8F> rc-local.service           loaded failed failed /etc/rc.d/rc.local Compatibility
<E2><97><8F> rngd.service               loaded failed failed Hardware RNG Entropy Gatherer Daemon
<E2><97><8F> systemd-remount-fs.service loaded failed failed Remount Root and Kernel File Systems
<E2><97><8F> systemd-sysctl.service     loaded failed failed Apply Kernel Variables

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

7 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

and

journalctl -xe --no-pager | grep 'Operation not permitted' | awk '{print $6,$7,$8,$9,$10,$11,$12,$13,$14,$15}' | sort | uniq
Failed to reset devices.list on /system.slice/console-getty.service: Operation not permitted 
Failed to reset devices.list on /system.slice/crond.service: Operation not permitted 
Failed to reset devices.list on /system.slice/csf.service: Operation not permitted 
Failed to reset devices.list on /system.slice/dbus.service: Operation not permitted 
Failed to reset devices.list on /system.slice/haveged.service: Operation not permitted 
Failed to reset devices.list on /system.slice/ip6tables.service: Operation not permitted 
Failed to reset devices.list on /system.slice/iptables.service: Operation not permitted 
Failed to reset devices.list on /system.slice/lfd.service: Operation not permitted 
Failed to reset devices.list on /system.slice/mariadb.service: Operation not permitted 
Failed to reset devices.list on /system.slice/memcached.service: Operation not permitted 
Failed to reset devices.list on /system.slice/network.service: Operation not permitted 
Failed to reset devices.list on /system.slice/nginx.service: Operation not permitted 
Failed to reset devices.list on /system.slice/ntpd.service: Operation not permitted 
Failed to reset devices.list on /system.slice/php-fpm.service: Operation not permitted 
Failed to reset devices.list on /system.slice/postfix.service: Operation not permitted 
Failed to reset devices.list on /system.slice/pure-ftpd.service: Operation not permitted 
Failed to reset devices.list on /system.slice/rc-local.service: Operation not permitted 
Failed to reset devices.list on /system.slice/rhel-domainname.service: Operation not permitted 
Failed to reset devices.list on /system.slice/rhel-readonly.service: Operation not permitted 
Failed to reset devices.list on /system.slice/rngd.service: Operation not permitted 
Failed to reset devices.list on /system.slice/rsyslog.service: Operation not permitted 
Failed to reset devices.list on /system.slice/sshd.service: Operation not permitted 
Failed to reset devices.list on /system.slice/sysstat.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-journal-flush.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-logind.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-random-seed.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-tmpfiles-clean.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-tmpfiles-setup-dev.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-tmpfiles-setup.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-udev-trigger.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-udevd.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-update-utmp-runlevel.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-update-utmp.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-user-sessions.service: Operation not permitted 
Failed to reset devices.list on /system.slice/tmp.mount: Operation not permitted 
Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted 
RNDADDENTROPY failed: Operation not permitted     
ntp_adjtime() failed: Operation not permitted     
pam_limits(crond:session): Could not set limit for 'nofile': Operation not permitted
7

next update to facebook rpm backported systemd 234 and restarting mariadb works - yay !

wget https://copr.fedorainfracloud.org/coprs/jsynacek/systemd-backports-for-centos-7/repo/epel-7/jsynacek-systemd-backports-for-centos-7-epel-7.repo -O /etc/yum.repos.d/jsynacek-systemd-centos-7.repo
yum -y update systemd

systemctl --version
systemd 234
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN default-hierarchy=hybrid

systemctl restart mariadb

and

systemctl status mariadb 
â—Ź mariadb.service - MariaDB 10.1.33 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf, openfileslimit.conf, protecthome.conf
   Active: active (running) since Tue 2018-06-05 20:38:33 UTC; 1min 37s ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 1222 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 963 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
  Process: 962 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
 Main PID: 1198 (mysqld)
   Status: "Taking your SQL requests now..."
    Tasks: 23 (limit: 4915)
   CGroup: /system.slice/mariadb.service
           └─1198 /usr/sbin/mysqld

Jun 05 20:38:31 centos75 systemd[1]: Starting MariaDB 10.1.33 database server...
Jun 05 20:38:32 centos75 mysqld[1198]: 2018-06-05 20:38:32 139906587511104 [Note] /usr/sbin/mysqld (mysqld 10.1.33-MariaDB) starting as process 1198 ...
Jun 05 20:38:33 centos75 systemd[1]: Started MariaDB 10.1.33 database server.

reduced number of failed units

systemctl list-units --failed
  UNIT                          LOAD   ACTIVE SUB    DESCRIPTION                                 
â—Ź sys-kernel-config.mount       loaded failed failed Kernel Configuration File System            
â—Ź haveged.service               loaded failed failed Entropy Daemon based on the HAVEGE algorithm
â—Ź rngd.service                  loaded failed failed Hardware RNG Entropy Gatherer Daemon        
â—Ź systemd-remount-fs.service    loaded failed failed Remount Root and Kernel File Systems        
â—Ź systemd-journald-audit.socket loaded failed failed Journal Audit Socket                        

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

5 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

though journalctl -xe reporting more entries

journalctl -xe --no-pager | grep 'Operation not permitted' | awk '{print $6,$7,$8,$9,$10,$11,$12,$13,$14,$15}' | sort | uniq
console-getty.service: Failed to reset devices.list: Operation not permitted  
console-getty.service: Failed to set invocation ID on control group /system.slice/console-getty.service,
crond.service: Failed to reset devices.list: Operation not permitted  
crond.service: Failed to set invocation ID on control group /system.slice/crond.service,
csf.service: Failed to reset devices.list: Operation not permitted  
csf.service: Failed to set invocation ID on control group /system.slice/csf.service,
dbus.service: Failed to reset devices.list: Operation not permitted  
dbus.service: Failed to set invocation ID on control group /system.slice/dbus.service,
dev-full.mount: Failed to reset devices.list: Operation not permitted  
dev-fuse.mount: Failed to reset devices.list: Operation not permitted  
dev-lxd.mount: Failed to reset devices.list: Operation not permitted  
dev-.lxd\x2dmounts.mount: Failed to reset devices.list: Operation not permitted  
dev-mqueue.mount: Failed to reset devices.list: Operation not permitted  
dev-net-tun.mount: Failed to reset devices.list: Operation not permitted  
dev-null.mount: Failed to reset devices.list: Operation not permitted  
dev-ptmx.mount: Failed to reset devices.list: Operation not permitted  
dev-random.mount: Failed to reset devices.list: Operation not permitted  
dev-tty.mount: Failed to reset devices.list: Operation not permitted  
dev-urandom.mount: Failed to reset devices.list: Operation not permitted  
dev-zero.mount: Failed to reset devices.list: Operation not permitted  
Failed to reset devices.list on /system.slice/console-getty.service: Operation not permitted 
Failed to reset devices.list on /system.slice/crond.service: Operation not permitted 
Failed to reset devices.list on /system.slice/csf.service: Operation not permitted 
Failed to reset devices.list on /system.slice/dbus.service: Operation not permitted 
Failed to reset devices.list on /system.slice/haveged.service: Operation not permitted 
Failed to reset devices.list on /system.slice/ip6tables.service: Operation not permitted 
Failed to reset devices.list on /system.slice/iptables.service: Operation not permitted 
Failed to reset devices.list on /system.slice/lfd.service: Operation not permitted 
Failed to reset devices.list on /system.slice/mariadb.service: Operation not permitted 
Failed to reset devices.list on /system.slice/memcached.service: Operation not permitted 
Failed to reset devices.list on /system.slice/network.service: Operation not permitted 
Failed to reset devices.list on /system.slice/nginx.service: Operation not permitted 
Failed to reset devices.list on /system.slice/ntpd.service: Operation not permitted 
Failed to reset devices.list on /system.slice/php-fpm.service: Operation not permitted 
Failed to reset devices.list on /system.slice/postfix.service: Operation not permitted 
Failed to reset devices.list on /system.slice/pure-ftpd.service: Operation not permitted 
Failed to reset devices.list on /system.slice/rc-local.service: Operation not permitted 
Failed to reset devices.list on /system.slice/rhel-domainname.service: Operation not permitted 
Failed to reset devices.list on /system.slice/rhel-readonly.service: Operation not permitted 
Failed to reset devices.list on /system.slice/rngd.service: Operation not permitted 
Failed to reset devices.list on /system.slice/rsyslog.service: Operation not permitted 
Failed to reset devices.list on /system.slice/sshd.service: Operation not permitted 
Failed to reset devices.list on /system.slice/sysstat.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-journal-flush.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-logind.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-random-seed.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-tmpfiles-clean.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-tmpfiles-setup-dev.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-tmpfiles-setup.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-udevd.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-udev-trigger.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-update-utmp-runlevel.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-update-utmp.service: Operation not permitted 
Failed to reset devices.list on /system.slice/systemd-user-sessions.service: Operation not permitted 
Failed to reset devices.list on /system.slice/tmp.mount: Operation not permitted 
Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted 
haveged.service: Failed to reset devices.list: Operation not permitted  
haveged.service: Failed to set invocation ID on control group /system.slice/haveged.service,
init.scope: Failed to reset devices.list: Operation not permitted  
ip6tables.service: Failed to reset devices.list: Operation not permitted  
ip6tables.service: Failed to set invocation ID on control group /system.slice/ip6tables.service,
iptables.service: Failed to reset devices.list: Operation not permitted  
iptables.service: Failed to set invocation ID on control group /system.slice/iptables.service,
lfd.service: Failed to reset devices.list: Operation not permitted  
lfd.service: Failed to set invocation ID on control group /system.slice/lfd.service,
mariadb.service: Failed to reset devices.list: Operation not permitted  
mariadb.service: Failed to set invocation ID on control group /system.slice/mariadb.service,
memcached.service: Failed to reset devices.list: Operation not permitted  
memcached.service: Failed to set invocation ID on control group /system.slice/memcached.service,
-.mount: Failed to reset devices.list: Operation not permitted  
network.service: Failed to reset devices.list: Operation not permitted  
network.service: Failed to set invocation ID on control group /system.slice/network.service,
nginx.service: Failed to reset devices.list: Operation not permitted  
nginx.service: Failed to set invocation ID on control group /system.slice/nginx.service,
ntp_adjtime() failed: Operation not permitted     
ntpd.service: Failed to reset devices.list: Operation not permitted  
ntpd.service: Failed to set invocation ID on control group /system.slice/ntpd.service,
pam_limits(crond:session): Could not set limit for 'nofile': Operation not permitted
php-fpm.service: Failed to reset devices.list: Operation not permitted  
php-fpm.service: Failed to set invocation ID on control group /system.slice/php-fpm.service,
postfix.service: Failed to reset devices.list: Operation not permitted  
postfix.service: Failed to set invocation ID on control group /system.slice/postfix.service,
proc-cpuinfo.mount: Failed to reset devices.list: Operation not permitted  
proc-diskstats.mount: Failed to reset devices.list: Operation not permitted  
proc-meminfo.mount: Failed to reset devices.list: Operation not permitted  
proc-stat.mount: Failed to reset devices.list: Operation not permitted  
proc-swaps.mount: Failed to reset devices.list: Operation not permitted  
proc-sys-fs-binfmt_misc.mount: Failed to reset devices.list: Operation not permitted  
proc-uptime.mount: Failed to reset devices.list: Operation not permitted  
pure-ftpd.service: Failed to reset devices.list: Operation not permitted  
pure-ftpd.service: Failed to set invocation ID on control group /system.slice/pure-ftpd.service,
rhel-domainname.service: Failed to reset devices.list: Operation not permitted  
rhel-domainname.service: Failed to set invocation ID on control group /system.slice/rhel-domainname.service,
rhel-readonly.service: Failed to reset devices.list: Operation not permitted  
rhel-readonly.service: Failed to set invocation ID on control group /system.slice/rhel-readonly.service,
RNDADDENTROPY failed: Operation not permitted     
rngd.service: Failed to reset devices.list: Operation not permitted  
rngd.service: Failed to set invocation ID on control group /system.slice/rngd.service,
rsyslog.service: Failed to reset devices.list: Operation not permitted  
rsyslog.service: Failed to set invocation ID on control group /system.slice/rsyslog.service,
session-c1.scope: Failed to reset devices.list: Operation not permitted  
session-c1.scope: Failed to set invocation ID on control group /user.slice/user-0.slice/session-c1.scope,
sshd.service: Failed to reset devices.list: Operation not permitted  
sshd.service: Failed to set invocation ID on control group /system.slice/sshd.service,
sys-fs-fuse-connections.mount: Failed to reset devices.list: Operation not permitted  
sys-kernel-debug.mount: Failed to reset devices.list: Operation not permitted  
sysstat.service: Failed to reset devices.list: Operation not permitted  
sysstat.service: Failed to set invocation ID on control group /system.slice/sysstat.service,
systemd-halt.service: Failed to reset devices.list: Operation not permitted  
systemd-halt.service: Failed to set invocation ID on control group /system.slice/systemd-halt.service,
systemd-journald.service: Failed to reset devices.list: Operation not permitted  
systemd-journal-flush.service: Failed to reset devices.list: Operation not permitted  
systemd-journal-flush.service: Failed to set invocation ID on control group /system.slice/systemd-journal-flush.service,
systemd-logind.service: Failed to reset devices.list: Operation not permitted  
systemd-logind.service: Failed to set invocation ID on control group /system.slice/systemd-logind.service,
systemd-random-seed.service: Failed to reset devices.list: Operation not permitted  
systemd-sysctl.service: Failed to reset devices.list: Operation not permitted  
systemd-sysctl.service: Failed to set invocation ID on control group /system.slice/systemd-sysctl.service,
systemd-tmpfiles-setup-dev.service: Failed to reset devices.list: Operation not permitted  
systemd-tmpfiles-setup-dev.service: Failed to set invocation ID on control group /system.slice/systemd-tmpfiles-setup-dev.service,
systemd-tmpfiles-setup.service: Failed to reset devices.list: Operation not permitted  
systemd-tmpfiles-setup.service: Failed to set invocation ID on control group /system.slice/systemd-tmpfiles-setup.service,
systemd-udevd.service: Failed to reset devices.list: Operation not permitted  
systemd-udevd.service: Failed to set invocation ID on control group /system.slice/systemd-udevd.service,
systemd-udev-trigger.service: Failed to reset devices.list: Operation not permitted  
systemd-udev-trigger.service: Failed to set invocation ID on control group /system.slice/systemd-udev-trigger.service,
systemd-update-utmp-runlevel.service: Failed to reset devices.list: Operation not permitted  
systemd-update-utmp-runlevel.service: Failed to set invocation ID on control group /system.slice/systemd-update-utmp-runlevel.service,
systemd-update-utmp.service: Failed to reset devices.list: Operation not permitted  
systemd-update-utmp.service: Failed to set invocation ID on control group /system.slice/systemd-update-utmp.service,
systemd-user-sessions.service: Failed to reset devices.list: Operation not permitted  
systemd-user-sessions.service: Failed to set invocation ID on control group /system.slice/systemd-user-sessions.service,
system.slice: Failed to reset devices.list: Operation not permitted  
system.slice: Failed to set invocation ID on control group /system.slice,
tmp.mount: Failed to reset devices.list: Operation not permitted  
tmp.mount: Failed to set invocation ID on control group /system.slice/tmp.mount,
user@0.service: Failed to reset devices.list: Operation not permitted  
user@0.service: Failed to set invocation ID on control group /user.slice/user-0.slice/user@0.service,
user-0.slice: Failed to reset devices.list: Operation not permitted  
user-0.slice: Failed to set invocation ID on control group /user.slice/user-0.slice,
user.slice: Failed to reset devices.list: Operation not permitted  
user.slice: Failed to set invocation ID on control group /user.slice,
8

They all look expected I think, the cgroup stuff is harmless, the devices cgroup is effectively non-functional for unprivileged containers, we could teach systemd to not even bother trying to set it up, but that’s harder than the current behavior which is to just ignore EPERM.

Looks like some unit attempted to add entropy to the kernel pool which was rejected (RNDADDENTROPY), that’s indeed not something a container should be allowed to do, if that caused anything useful to fail startup, then a bug report should be filed against it to handle unprivileged containers.

ntp_adjtime() is in the same camp, containers aren’t allowed to modify the system time as the clock is kernel-wide.

And lastly, the same applies to pam_limits where a container is allowed to reduce its ulimit/prlimit values (like nofile) but not raise it as that’d be a potential way of causing denial of service attacks against the host.

1 Like
9

Cheers thanks for clarification. I’ll use systemd-detect-virt = lxc to add checks into my LEMP stack installer to skip setting up haveged and ntpd at least :slight_smile:

10

Ok, the set of failed units make sense I think:

  • sys-kernel-config: Probably tried to mount debugfs or a similarly restricted filesystem
  • haveged: This would attempt to add entropy to the global kernel pool
  • rngd: Same as haveged but feeding CPU hardware entropy instead
  • systemd-journald-audit: Tried to access the kernel audit facility which is restricted to real root

The only one that probably shouldn’t have failed is systemd-remount-fs but that may depend on /etc/fstab or whatever other mount units are around. A filesystem remount is certainly allowed inside containers but maybe that unit failed because it couldn’t find the block device?

1 Like
11

within centos75 container

df -hT
Filesystem                  Type      Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-root ext4       79G   15G   60G  20% /
none                        tmpfs     492K     0  492K   0% /dev
udev                        devtmpfs  7.9G     0  7.9G   0% /dev/tty
tmpfs                       tmpfs     100K     0  100K   0% /dev/lxd
tmpfs                       tmpfs     100K     0  100K   0% /dev/.lxd-mounts
tmpfs                       tmpfs     7.9G     0  7.9G   0% /dev/shm
tmpfs                       tmpfs     7.9G   96K  7.9G   1% /run
tmpfs                       tmpfs     7.9G     0  7.9G   0% /sys/fs/cgroup
tmpfs                       tmpfs     7.9G     0  7.9G   0% /tmp

in lxd host on Ubuntu 18.04 LTS KVM VPS

df -hT  
Filesystem                  Type      Size  Used Avail Use% Mounted on
udev                        devtmpfs  7.9G     0  7.9G   0% /dev
tmpfs                       tmpfs     1.6G  936K  1.6G   1% /run
/dev/mapper/ubuntu--vg-root ext4       79G   15G   60G  20% /
tmpfs                       tmpfs     7.9G     0  7.9G   0% /dev/shm
tmpfs                       tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                       tmpfs     7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda1                   ext2      472M  128M  320M  29% /boot
/dev/loop0                  squashfs   87M   87M     0 100% /snap/core/4650
/dev/loop1                  squashfs   55M   55M     0 100% /snap/lxd/7412
tmpfs                       tmpfs     1.6G     0  1.6G   0% /run/user/0
12

What’s in /etc/fstab in the container?

The output above suggests that the container may have tried to remount /dev/mapper/ubuntu–vg-root which it obviously can’t see since that’s a host path, it’s not really something to worry about other than just making log files a bit cleaner :slight_smile:

1 Like
13

i see within container

cat /etc/fstab 
/dev/root               /                       rootfs   defaults        0 0
/swapfile swap swap defaults 0 0
tmpfs /tmp tmpfs rw,noexec,nosuid 0 0

looks like setting up swap file and tmpfs in container is another set of items to skip ?

14

seems back to the same problem ??

-- Unit mariadb.service has finished shutting down.
Jun 20 19:38:22 centos75-2 systemd[1]: mariadb.service: Unit entered failed state.
Jun 20 19:38:22 centos75-2 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Jun 20 19:38:22 centos75-2 systemd[1]: mariadb.service: Failed to reset devices.list: Operation not permitted
Jun 20 19:38:22 centos75-2 systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jun 20 19:38:22 centos75-2 systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jun 20 19:38:22 centos75-2 systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jun 20 19:38:22 centos75-2 systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jun 20 19:38:22 centos75-2 systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jun 20 19:38:22 centos75-2 systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jun 20 19:38:22 centos75-2 systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jun 20 19:38:22 centos75-2 systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jun 20 19:38:22 centos75-2 systemd[1]: mariadb.service: Failed to set invocation ID on control group /system.slice/mariadb.service, ignoring: Operation not permitted
Jun 20 19:38:22 centos75-2 systemd[1]: Starting MariaDB 10.1.33 database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mariadb.service has begun starting up.
Jun 20 19:38:23 centos75-2 mysqld[1170]: 2018-06-20 19:38:23 139984637413696 [Note] /usr/sbin/mysqld (mysqld 10.1.33-MariaDB) starting as process 1170 ...
Jun 20 19:38:23 centos75-2 mysqld[1170]: 2018-06-20 19:38:23 139984637413696 [Note] InnoDB: Using mutexes to ref count buffer pool pages
Jun 20 19:38:23 centos75-2 mysqld[1170]: 2018-06-20 19:38:23 139984637413696 [Note] InnoDB: The InnoDB memory heap is disabled
Jun 20 19:38:23 centos75-2 mysqld[1170]: 2018-06-20 19:38:23 139984637413696 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Jun 20 19:38:23 centos75-2 mysqld[1170]: 2018-06-20 19:38:23 139984637413696 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
Jun 20 19:38:23 centos75-2 mysqld[1170]: 2018-06-20 19:38:23 139984637413696 [Note] InnoDB: Compressed tables use zlib 1.2.7
Jun 20 19:38:23 centos75-2 mysqld[1170]: 2018-06-20 19:38:23 139984637413696 [Note] InnoDB: Using Linux native AIO
Jun 20 19:38:23 centos75-2 mysqld[1170]: 2018-06-20 19:38:23 139984637413696 [Note] InnoDB: Using SSE crc32 instructions
Jun 20 19:38:23 centos75-2 mysqld[1170]: 2018-06-20 19:38:23 139984637413696 [Note] InnoDB: Initializing buffer pool, size = 4.0G
Jun 20 19:38:23 centos75-2 mysqld[1170]: 2018-06-20 19:38:23 139984637413696 [Note] InnoDB: Completed initialization of buffer pool
Jun 20 19:38:23 centos75-2 mysqld[1170]: 2018-06-20 19:38:23 139984637413696 [Note] InnoDB: Highest supported file format is Barracuda.
Jun 20 19:38:24 centos75-2 mysqld[1170]: 2018-06-20 19:38:24 139984637413696 [Note] InnoDB: 128 rollback segment(s) are active.
Jun 20 19:38:24 centos75-2 mysqld[1170]: 2018-06-20 19:38:24 139984637413696 [Note] InnoDB: Waiting for purge to start
Jun 20 19:38:24 centos75-2 mysqld[1170]: 2018-06-20 19:38:24 139984637413696 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.39-83.1 started; log sequence number 1621582
Jun 20 19:38:24 centos75-2 mysqld[1170]: 2018-06-20 19:38:24 139977811781376 [Note] InnoDB: Dumping buffer pool(s) not yet started
Jun 20 19:38:24 centos75-2 mysqld[1170]: 2018-06-20 19:38:24 139984637413696 [Note] Plugin 'FEEDBACK' is disabled.
Jun 20 19:38:24 centos75-2 mysqld[1170]: 2018-06-20 19:38:24 139984637413696 [Note] Recovering after a crash using tc.log
Jun 20 19:38:24 centos75-2 mysqld[1170]: 2018-06-20 19:38:24 139984637413696 [ERROR] Can't init tc log
Jun 20 19:38:24 centos75-2 mysqld[1170]: 2018-06-20 19:38:24 139984637413696 [ERROR] Aborting
Jun 20 19:38:26 centos75-2 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Jun 20 19:38:26 centos75-2 systemd[1]: Failed to start MariaDB 10.1.33 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
15

whoops removing tc.log and restarting mysql was the fix :slight_smile:

16

Hello @stgraber,

I am trying to install an Ubuntu package called privacyidea-apache2 into a LXD container and have received an error that I think is related to the above comment you made back in 2018. As part of the final configuration, the apt package is apparently calling rngd and the system is failing.

Oct 25 11:14:55 idea rngd: RNDADDENTROPY failed: Operation not permitted

Can you think of a work around that would allow me to complete the installation of this package, given that containers apparently should not be able to add entropy to the kernel?

I really want this application in a container if possible.

Thanks.
John

17

I may have figured out a work around. Using info about privileged containers from this post:

it appears that I can make the container privileged:

lxc config set [container] security.privileged true

and then complete the installation that requires the ability to add entropy to the kernel pool. I’m assuming it would be wise to remove the privileged status after the install is complete, so I’ll do that and continue to experiment.

If anyone can comment on whether this action is unwise, I would appreciate hear it.