Escape attempt LXD Bug ? / strange process tree with different map ID / can not allocate memory

One LXD Hostmachine becomes unresponsible because it can not create new processes.

Everything results in:

-bash: fork: Cannot allocate memory

I was lucky enough to catch a moment in time, where i was able to observe this behaviour and still being able to spawn processes.

This is what i have found:

top - 12:26:37 up 1 day, 52 min,  1 user,  load average: 129.68, 128.03, 124.78
Tasks: 15989 total,   1 running, 1150 sleeping,   0 stopped, 14838 zombie

( 14838 zombies, rising numbers )

This are the zombies:

1000105   8331  0.0  0.0      0     0 ?        Z    09:16   0:00      \_ [sshd] <defunct>
1000105   8347  0.0  0.0      0     0 ?        Z    09:16   0:00      \_ [sshd] <defunct>
1000105   9313  0.0  0.0      0     0 ?        Z    09:16   0:00      \_ [sshd] <defunct>
1000105   9823  0.0  0.0      0     0 ?        Z    09:16   0:00      \_ [sshd] <defunct>
1000105  10696  0.0  0.0      0     0 ?        Z    09:16   0:00      \_ [sshd] <defunct>
1000105  11137  0.0  0.0      0     0 ?        Z    09:16   0:00      \_ [sshd] <defunct>
1000105  12463  0.0  0.0      0     0 ?        Z    09:17   0:00      \_ [sshd] <defunct>
1000105  12906  0.0  0.0      0     0 ?        Z    09:17   0:00      \_ [sshd] <defunct>
1000105  13749  0.0  0.0      0     0 ?        Z    09:17   0:00      \_ [sshd] <defunct>
1000105  13758  0.0  0.0      0     0 ?        Z    09:17   0:00      \_ [sshd] <defunct>
1000105  14719  0.0  0.0      0     0 ?        Z    09:17   0:00      \_ [sshd] <defunct>
1000105  15378  0.0  0.0      0     0 ?        Z    09:17   0:00      \_ [sshd] <defunct>
1000105  15380  0.0  0.0      0     0 ?        Z    09:17   0:00      \_ [sshd] <defunct>
1000105  17725  0.0  0.0      0     0 ?        Z    09:18   0:00      \_ [sshd] <defunct>
1000105  18544  0.0  0.0      0     0 ?        Z    09:18   0:00      \_ [sshd] <defunct>
1000105  19898  0.0  0.0      0     0 ?        Z    09:19   0:00      \_ [sshd] <defunct>
1000105  20160  0.0  0.0      0     0 ?        Z    09:19   0:00      \_ [sshd] <defunct>

So i was going to check which LXD Container this all belongs to.

Checking /proc/$pid the cgroups file or any other files giving useful informations where

readlink failed

Ok so there was no way ( with my knowledge ) to get out any intresting informations there.

Before every command resulted in a can not allocate memory, i took the processtree and while searching through the 15k lines, i found this:

root     24994  0.0  0.0 1358416 27968 ?       Ss   May27   0:00 [lxc monitor] /var/snap/lxd/common/lxd/containers lxd313
1000000  25012  0.0  0.0 225152  5964 ?        Ss   May27   0:03  \_ /sbin/init
1000000  27581  0.0  0.0 256800 126388 ?       Ss   May27   0:32      \_ /lib/systemd/systemd-journald
1000000  29000  0.0  0.0  42092  1704 ?        Ss   May27   0:00      \_ /lib/systemd/systemd-udevd
1000100  30207  0.0  0.0  71848  3380 ?        Ss   May27   0:00      \_ /lib/systemd/systemd-networkd
1000101  31541  0.0  0.0  70628  2988 ?        Ss   May27   0:00      \_ /lib/systemd/systemd-resolved
1000103  32306  0.0  0.0  49916  2204 ?        Ss   May27   0:01      \_ /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
1000102    712  0.0  0.0 193388  3072 ?        Ssl  May27   0:06      \_ /usr/sbin/rsyslogd -n
1000108   1720  0.1  0.0  55528  8072 ?        Ssl  May27   1:40      \_ /usr/bin/redis-server 127.0.0.1:6379
1000000   2206  0.0  0.0 170420 12860 ?        Ssl  May27   0:00      \_ /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
1000000   2527  0.0  0.0 446148 16344 ?        Ss   May27   0:03      \_ php-fpm: master process (/etc/php/7.2/fpm/php-fpm.conf)
1000113  12248  0.0  0.0 448968 12208 ?        S    May27   0:01      |   \_ php-fpm: pool www
1000113  12249  0.0  0.0 448968 12044 ?        S    May27   0:01      |   \_ php-fpm: pool www
1000000   2889  0.0  0.0  70604  3216 ?        Ss   May27   0:00      \_ /lib/systemd/systemd-logind
1000000   5967  0.0  0.0  31308  1516 ?        Ss   May27   0:01      \_ /usr/sbin/cron -f
1000000  20669  0.0  0.0  58788  1652 ?        S    05:20   0:00      |   \_ /usr/sbin/CRON -f
1000113  20670  0.0  0.0   4616   660 ?        Ss   05:20   0:00      |       \_ /bin/sh -c [ -e /usr/sbin/fwconsole ] && sleep $((RANDOM%30)) && /usr/sbin/fwconsole job --run --quiet 2>&1 > /dev/null
1000113  20852  0.0  0.0 316636 32596 ?        S    05:20   0:00      |           \_ php /usr/sbin/fwconsole job --run --quiet
1000109   9244  0.0  0.0 736340 40676 ?        S    May27   0:22      \_ /usr/lib/postgresql/10/bin/postgres -D /var/lib/postgresql/10/main -c config_file=/etc/postgresql/10/main/postgresql.conf
1000109  27956  0.0  0.0 736476  8140 ?        Ss   May27   0:00      |   \_ postgres: 10/main: checkpointer process   
1000109  27957  0.0  0.0 736340  7096 ?        Ss   May27   0:01      |   \_ postgres: 10/main: writer process   
1000109  27958  0.0  0.0 736340 18976 ?        Ss   May27   0:01      |   \_ postgres: 10/main: wal writer process   
1000109  27959  0.0  0.0 736904  3544 ?        Ss   May27   0:05      |   \_ postgres: 10/main: autovacuum launcher process   
1000109  27960  0.0  0.0 175508  2764 ?        Ss   May27   0:08      |   \_ postgres: 10/main: stats collector process   
1000109  27961  0.0  0.0 736624  3040 ?        Ss   May27   0:00      |   \_ postgres: 10/main: bgworker: logical replication launcher   
1000109  24470  0.0  0.0 737532  8244 ?        Ss   May27   0:00      |   \_ postgres: 10/main: openslides-spd-nw openslides-spd-nw 127.0.0.1(51932) idle
1000109  24493  0.0  0.0 737532  8244 ?        Ss   May27   0:00      |   \_ postgres: 10/main: openslides-generali-kbr openslides-generali-kbr 127.0.0.1(51936) idle
1000109  24927  0.0  0.0 737532  8188 ?        Ss   May27   0:00      |   \_ postgres: 10/main: openslides-gemeinderat-nw openslides-gemeinderat-nw 127.0.0.1(51944) idle
1000109  25118  0.0  0.0 737532  8208 ?        Ss   May27   0:00      |   \_ postgres: 10/main: openslides-gdis-br-hh openslides-gdis-br-hh 127.0.0.1(51952) idle
1000109  25243  0.0  0.0 737532  8232 ?        Ss   May27   0:00      |   \_ postgres: 10/main: openslides-gdis-gbr openslides-gdis-gbr 127.0.0.1(51962) idle
1000000   9374  0.0  0.0  54968 13960 ?        Ss   May27   0:04      \_ /usr/bin/perl -wT /usr/sbin/munin-node
1000000  13831  0.1  0.0 116888  5176 ?        Sl   May27   2:30      \_ /usr/bin/monit -c /etc/monit/monitrc
1000000  20745  0.0  0.0  15944   936 ?        Ss+  May27   0:00      \_ /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 linux
1000000  21899  0.0  0.0  72288  3384 ?        Ss   May27   0:01      \_ /usr/sbin/sshd -D
1000000   1572  0.0  0.0 105676  3972 ?        Ss   12:25   0:00      |   \_ sshd: root [priv]
1000105   2005  0.0  0.0  72288  1424 ?        S    12:25   0:00      |   |   \_ sshd: root [net]
1000000  12698  0.0  0.0 105676  4076 ?        Ss   12:25   0:00      |   \_ sshd: root [priv]
1000105  12852  0.0  0.0  72288  1428 ?        S    12:25   0:00      |       \_ sshd: root [net]
1000000  22099  0.0  0.0  36248  1440 ?        Ss   May27   0:00      \_ nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
1000033  22116  0.0  0.0  42208  5968 ?        S    May27   0:08      |   \_ nginx: worker process
1000033  22117  0.0  0.0  40896  4472 ?        S    May27   0:00      |   \_ nginx: worker process
1000033  22118  0.0  0.0  40624  3700 ?        S    May27   0:00      |   \_ nginx: worker process
1000033  22119  0.0  0.0  40624  2252 ?        S    May27   0:00      |   \_ nginx: worker process
1000033  22121  0.0  0.0  40624  2252 ?        S    May27   0:00      |   \_ nginx: worker process
1000033  22122  0.0  0.0  40624  2252 ?        S    May27   0:00      |   \_ nginx: worker process
1000033  22123  0.0  0.0  40624  2252 ?        S    May27   0:00      |   \_ nginx: worker process
1000033  22124  0.0  0.0  40624  2252 ?        S    May27   0:00      |   \_ nginx: worker process
1000033  22125  0.0  0.0  40624  2252 ?        S    May27   0:00      |   \_ nginx: worker process
1000033  22127  0.0  0.0  40624  2252 ?        S    May27   0:00      |   \_ nginx: worker process
1000033  22421  0.0  0.0  40624  2252 ?        S    May27   0:00      |   \_ nginx: worker process
1000033  22422  0.0  0.0  40624  2252 ?        S    May27   0:00      |   \_ nginx: worker process
1000114  25846  0.1  0.1 736236 239288 ?       Ssl  May27   1:55      \_ /usr/sbin/mysqld
1000113  14373  0.5  0.0 7112116 67680 ?       Ssl  May27   8:29      \_ /usr/sbin/asterisk -g -f -U asterisk
1000000   1290  0.0  0.0  76532  4088 ?        Ss   May27   0:00      \_ /lib/systemd/systemd --user
1000000   1662  0.0  0.0 256832  2344 ?        S    May27   0:00      |   \_ (sd-pam)
1065534  22231  0.0  0.0  11452   724 ?        D    05:20   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  24212  0.0  0.0  11452   724 ?        D    05:21   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  26238  0.0  0.0  22688  1632 ?        D    05:22   0:00      \_ awk /processes/ {print $2} /proc/stat
1065534  11274  0.0  0.0  11452   728 ?        D    05:30   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  13089  0.0  0.0  11452   720 ?        D    05:31   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  15299  0.0  0.0  22688  1632 ?        D    05:32   0:00      \_ awk /processes/ {print $2} /proc/stat
1065534   1156  0.0  0.0      0     0 ?        Zs   05:40   0:00      \_ [cpu] <defunct>
1065534   1157  0.0  0.0  11452   724 ?        D    05:40   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534   3134  0.0  0.0      0     0 ?        Zs   05:41   0:00      \_ [cpu] <defunct>
1065534   3135  0.0  0.0  11452   724 ?        D    05:41   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534   5775  0.0  0.0      0     0 ?        Zs   05:42   0:00      \_ [forks] <defunct>
1065534   5776  0.0  0.0  22688  1632 ?        D    05:42   0:00      \_ awk /processes/ {print $2} /proc/stat
1000111   7873  0.0  0.0      0     0 ?        ZN   05:43   0:03      \_ [munin-html] <defunct>
1000111   7874  0.0  0.0      0     0 ?        ZN   05:43   0:09      \_ [munin-html] <defunct>
1065534  21650  0.0  0.0      0     0 ?        Zs   05:50   0:00      \_ [cpu] <defunct>
1065534  21651  0.0  0.0  11452   720 ?        D    05:50   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105  22091  0.0  0.0      0     0 ?        Z    05:50   0:00      \_ [sshd] <defunct>
1065534  23554  0.0  0.0      0     0 ?        Zs   05:51   0:00      \_ [cpu] <defunct>
1065534  23555  0.0  0.0  11452   720 ?        D    05:51   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105  24507  0.0  0.0      0     0 ?        Z    05:51   0:00      \_ [sshd] <defunct>
1065534  25639  0.0  0.0      0     0 ?        Zs   05:52   0:00      \_ [forks] <defunct>
1065534  25640  0.0  0.0  22688  1632 ?        D    05:52   0:00      \_ awk /processes/ {print $2} /proc/stat
1000105  25835  0.0  0.0      0     0 ?        Z    05:52   0:00      \_ [sshd] <defunct>
1000111  28250  0.0  0.0      0     0 ?        ZN   05:53   0:03      \_ [munin-html] <defunct>
1000111  28251  0.0  0.0      0     0 ?        ZN   05:53   0:09      \_ [munin-html] <defunct>
1000105   5912  0.0  0.0      0     0 ?        Z    05:57   0:00      \_ [sshd] <defunct>
1065534  10999  0.0  0.0      0     0 ?        Zs   06:00   0:00      \_ [cpu] <defunct>
1065534  11000  0.0  0.0  11452   724 ?        D    06:00   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  13030  0.0  0.0      0     0 ?        Zs   06:01   0:00      \_ [cpu] <defunct>
1065534  13031  0.0  0.0  11452   720 ?        D    06:01   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105  14821  0.0  0.0      0     0 ?        Z    06:01   0:00      \_ [sshd] <defunct>
1065534  15342  0.0  0.0      0     0 ?        Zs   06:02   0:00      \_ [forks] <defunct>
1065534  15343  0.0  0.0  22688  1632 ?        D    06:02   0:00      \_ awk /processes/ {print $2} /proc/stat
1000111  17687  0.0  0.0      0     0 ?        ZN   06:03   0:03      \_ [munin-html] <defunct>
1000111  17688  0.0  0.0      0     0 ?        ZN   06:03   0:09      \_ [munin-html] <defunct>
1000105  27122  0.0  0.0      0     0 ?        Z    06:07   0:00      \_ [sshd] <defunct>
1065534   2548  0.0  0.0      0     0 ?        Zs   06:10   0:00      \_ [cpu] <defunct>
1065534   2549  0.0  0.0  11452   724 ?        D    06:10   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534   5014  0.0  0.0      0     0 ?        Zs   06:11   0:00      \_ [cpu] <defunct>
1065534   5015  0.0  0.0  11452   720 ?        D    06:11   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534   7367  0.0  0.0      0     0 ?        Zs   06:12   0:00      \_ [forks] <defunct>
1065534   7368  0.0  0.0  22688  1632 ?        D    06:12   0:00      \_ awk /processes/ {print $2} /proc/stat
1000111   9987  0.0  0.0      0     0 ?        ZN   06:13   0:03      \_ [munin-html] <defunct>
1000111   9988  0.0  0.0      0     0 ?        ZN   06:13   0:09      \_ [munin-html] <defunct>
1000105  20069  0.0  0.0      0     0 ?        Z    06:17   0:00      \_ [sshd] <defunct>
1065534  25909  0.0  0.0      0     0 ?        Zs   06:20   0:00      \_ [cpu] <defunct>
1065534  25911  0.0  0.0  11452   724 ?        D    06:20   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  28198  0.0  0.0      0     0 ?        Zs   06:21   0:00      \_ [cpu] <defunct>
1065534  28199  0.0  0.0  11452   720 ?        D    06:21   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105  30540  0.0  0.0      0     0 ?        Z    06:22   0:00      \_ [sshd] <defunct>
1065534  30657  0.0  0.0      0     0 ?        Zs   06:22   0:00      \_ [forks] <defunct>
1065534  30658  0.0  0.0  22688  1632 ?        D    06:22   0:00      \_ awk /processes/ {print $2} /proc/stat
1000111    448  0.0  0.0      0     0 ?        ZN   06:23   0:03      \_ [munin-html] <defunct>
1000111    449  0.0  0.0      0     0 ?        ZN   06:23   0:09      \_ [munin-html] <defunct>
1000105   2996  0.0  0.0      0     0 ?        Z    06:24   0:00      \_ [sshd] <defunct>
1000105  12843  0.0  0.0      0     0 ?        Z    06:27   0:00      \_ [sshd] <defunct>
1065534  18146  0.0  0.0      0     0 ?        Zs   06:30   0:00      \_ [cpu] <defunct>
1065534  18147  0.0  0.0  11452   720 ?        D    06:30   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  20561  0.0  0.0      0     0 ?        Zs   06:31   0:00      \_ [cpu] <defunct>
1065534  20562  0.0  0.0  11452   724 ?        D    06:31   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  22861  0.0  0.0      0     0 ?        Zs   06:32   0:00      \_ [forks] <defunct>
1065534  22862  0.0  0.0  22688  1632 ?        D    06:32   0:00      \_ awk /processes/ {print $2} /proc/stat
1000111  26461  0.0  0.0      0     0 ?        ZN   06:33   0:03      \_ [munin-html] <defunct>
1000111  26462  0.0  0.0      0     0 ?        ZN   06:33   0:09      \_ [munin-html] <defunct>
1000105  27491  0.0  0.0      0     0 ?        Z    06:33   0:00      \_ [sshd] <defunct>
1000105  31058  0.0  0.0      0     0 ?        Z    06:35   0:00      \_ [sshd] <defunct>
1000105    627  0.0  0.0      0     0 ?        Z    06:35   0:00      \_ [sshd] <defunct>
1000105   5277  0.0  0.0      0     0 ?        Z    06:37   0:00      \_ [sshd] <defunct>
1065534  11693  0.0  0.0      0     0 ?        Zs   06:40   0:00      \_ [cpu] <defunct>
1065534  11694  0.0  0.0  11452   720 ?        D    06:40   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  13870  0.0  0.0      0     0 ?        Zs   06:41   0:00      \_ [cpu] <defunct>
1065534  13871  0.0  0.0  11452   724 ?        D    06:41   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  16346  0.0  0.0      0     0 ?        Zs   06:42   0:00      \_ [forks] <defunct>
1065534  16347  0.0  0.0  22688  1632 ?        D    06:42   0:00      \_ awk /processes/ {print $2} /proc/stat
1000105  16513  0.0  0.0      0     0 ?        Z    06:42   0:00      \_ [sshd] <defunct>
1000111  19095  0.0  0.0      0     0 ?        ZN   06:43   0:03      \_ [munin-html] <defunct>
1000111  19096  0.0  0.0      0     0 ?        ZN   06:43   0:09      \_ [munin-html] <defunct>
1000105  29847  0.0  0.0      0     0 ?        Z    06:47   0:00      \_ [sshd] <defunct>
1000105  30288  0.0  0.0      0     0 ?        Z    06:47   0:00      \_ [sshd] <defunct>
1065534   2679  0.0  0.0      0     0 ?        Zs   06:50   0:00      \_ [cpu] <defunct>
1065534   2681  0.0  0.0  11452   720 ?        D    06:50   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534   5757  0.0  0.0      0     0 ?        Zs   06:51   0:00      \_ [cpu] <defunct>
1065534   5758  0.0  0.0  11452   724 ?        D    06:51   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105   7769  0.0  0.0      0     0 ?        Z    06:52   0:00      \_ [sshd] <defunct>
1065534   8176  0.0  0.0      0     0 ?        Zs   06:52   0:00      \_ [forks] <defunct>
1065534   8177  0.0  0.0  22688  1632 ?        D    06:52   0:00      \_ awk /processes/ {print $2} /proc/stat
1000105  10739  0.0  0.0      0     0 ?        Z    06:53   0:00      \_ [sshd] <defunct>
1000111  10747  0.0  0.0      0     0 ?        ZN   06:53   0:03      \_ [munin-html] <defunct>
1000111  10748  0.0  0.0      0     0 ?        ZN   06:53   0:09      \_ [munin-html] <defunct>
1000105  11180  0.0  0.0      0     0 ?        Z    06:53   0:00      \_ [sshd] <defunct>
1000105  23333  0.0  0.0      0     0 ?        Z    06:58   0:00      \_ [sshd] <defunct>
1000105  23334  0.0  0.0      0     0 ?        Z    06:58   0:00      \_ [sshd] <defunct>
1065534  28074  0.0  0.0      0     0 ?        Zs   07:00   0:00      \_ [cpu] <defunct>
1065534  28075  0.0  0.0  11452   728 ?        D    07:00   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105  30025  0.0  0.0      0     0 ?        Z    07:00   0:00      \_ [sshd] <defunct>
1065534  30541  0.0  0.0      0     0 ?        Zs   07:01   0:00      \_ [cpu] <defunct>
1065534  30542  0.0  0.0  11452   720 ?        D    07:01   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105  32238  0.0  0.0      0     0 ?        Z    07:01   0:00      \_ [sshd] <defunct>
1065534    456  0.0  0.0      0     0 ?        Zs   07:02   0:00      \_ [forks] <defunct>
1065534    632  0.0  0.0  22688  1632 ?        D    07:02   0:00      \_ awk /processes/ {print $2} /proc/stat
1000111   3816  0.0  0.0      0     0 ?        ZN   07:03   0:03      \_ [munin-html] <defunct>
1000111   3817  0.0  0.0      0     0 ?        ZN   07:03   0:09      \_ [munin-html] <defunct>
1000105  22143  0.0  0.0      0     0 ?        Z    07:10   0:00      \_ [sshd] <defunct>
1065534  22363  0.0  0.0      0     0 ?        Zs   07:10   0:00      \_ [cpu] <defunct>
1065534  22364  0.0  0.0  11452   724 ?        D    07:10   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  24808  0.0  0.0      0     0 ?        Zs   07:11   0:00      \_ [cpu] <defunct>
1065534  24809  0.0  0.0  11452   724 ?        D    07:11   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105  26684  0.0  0.0      0     0 ?        Z    07:12   0:00      \_ [sshd] <defunct>
1065534  27077  0.0  0.0      0     0 ?        Zs   07:12   0:00      \_ [forks] <defunct>
1065534  27078  0.0  0.0  22688  1632 ?        D    07:12   0:00      \_ awk /processes/ {print $2} /proc/stat
1000105  28308  0.0  0.0      0     0 ?        Z    07:12   0:00      \_ [sshd] <defunct>
1000111  29879  0.0  0.0      0     0 ?        ZN   07:13   0:03      \_ [munin-html] <defunct>
1000111  29880  0.0  0.0      0     0 ?        ZN   07:13   0:09      \_ [munin-html] <defunct>
1000105  30492  0.0  0.0      0     0 ?        Z    07:13   0:00      \_ [sshd] <defunct>
1000105    995  0.0  0.0      0     0 ?        Z    07:14   0:00      \_ [sshd] <defunct>
1000105   2510  0.0  0.0      0     0 ?        Z    07:15   0:00      \_ [sshd] <defunct>
1000105   2849  0.0  0.0      0     0 ?        Z    07:15   0:00      \_ [sshd] <defunct>
1000105   6118  0.0  0.0      0     0 ?        Z    07:16   0:00      \_ [sshd] <defunct>
1000105  15531  0.0  0.0      0     0 ?        Z    07:20   0:00      \_ [sshd] <defunct>
1065534  15697  0.0  0.0      0     0 ?        Zs   07:20   0:00      \_ [cpu] <defunct>
1065534  15698  0.0  0.0  11452   724 ?        D    07:20   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105  17064  0.0  0.0      0     0 ?        Z    07:20   0:00      \_ [sshd] <defunct>
1000105  17816  0.0  0.0      0     0 ?        Z    07:20   0:00      \_ [sshd] <defunct>
1065534  18401  0.0  0.0      0     0 ?        Zs   07:21   0:00      \_ [cpu] <defunct>
1065534  18402  0.0  0.0  11452   724 ?        D    07:21   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  20849  0.0  0.0      0     0 ?        Zs   07:22   0:00      \_ [forks] <defunct>
1065534  20850  0.0  0.0  22688  1632 ?        D    07:22   0:00      \_ awk /processes/ {print $2} /proc/stat
1000111  23490  0.0  0.0      0     0 ?        ZN   07:23   0:03      \_ [munin-html] <defunct>
1000111  23491  0.0  0.0      0     0 ?        ZN   07:23   0:09      \_ [munin-html] <defunct>
1000105  32301  0.0  0.0      0     0 ?        Z    07:26   0:00      \_ [sshd] <defunct>
1000105    583  0.0  0.0      0     0 ?        Z    07:26   0:00      \_ [sshd] <defunct>
1065534   9059  0.0  0.0      0     0 ?        Zs   07:30   0:00      \_ [cpu] <defunct>
1065534   9060  0.0  0.0  11452   724 ?        D    07:30   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  11627  0.0  0.0      0     0 ?        Zs   07:31   0:00      \_ [cpu] <defunct>
1065534  11629  0.0  0.0  11452   724 ?        D    07:31   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105  14275  0.0  0.0      0     0 ?        Z    07:32   0:00      \_ [sshd] <defunct>
1065534  14431  0.0  0.0      0     0 ?        Zs   07:32   0:00      \_ [forks] <defunct>
1065534  14432  0.0  0.0  22688  1636 ?        D    07:32   0:00      \_ awk /processes/ {print $2} /proc/stat
1000111  17552  0.0  0.0      0     0 ?        ZN   07:33   0:03      \_ [munin-html] <defunct>
1000111  17553  0.0  0.0      0     0 ?        ZN   07:33   0:09      \_ [munin-html] <defunct>
1000105   2513  0.0  0.0      0     0 ?        Z    07:39   0:00      \_ [sshd] <defunct>
1065534   4807  0.0  0.0      0     0 ?        Zs   07:40   0:00      \_ [cpu] <defunct>
1065534   4808  0.0  0.0  11452   724 ?        D    07:40   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534   7285  0.0  0.0      0     0 ?        Zs   07:41   0:00      \_ [cpu] <defunct>
1065534   7286  0.0  0.0  11452   720 ?        D    07:41   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105   8036  0.0  0.0      0     0 ?        Z    07:41   0:00      \_ [sshd] <defunct>
1000105   9753  0.0  0.0      0     0 ?        Z    07:42   0:00      \_ [sshd] <defunct>
1065534  10011  0.0  0.0      0     0 ?        Zs   07:42   0:00      \_ [forks] <defunct>
1065534  10013  0.0  0.0  22688  1632 ?        D    07:42   0:00      \_ awk /processes/ {print $2} /proc/stat
1000105  11911  0.0  0.0      0     0 ?        Z    07:42   0:00      \_ [sshd] <defunct>
1000111  13516  0.0  0.0      0     0 ?        ZN   07:43   0:03      \_ [munin-html] <defunct>
1000111  13518  0.0  0.0      0     0 ?        ZN   07:43   0:09      \_ [munin-html] <defunct>
1000105  14735  0.0  0.0      0     0 ?        Z    07:43   0:00      \_ [sshd] <defunct>
1000105  29836  0.0  0.0      0     0 ?        Z    07:49   0:00      \_ [sshd] <defunct>
1000105  30710  0.0  0.0      0     0 ?        Z    07:49   0:00      \_ [sshd] <defunct>
1000105  31308  0.0  0.0      0     0 ?        Z    07:49   0:00      \_ [sshd] <defunct>
1000105  31970  0.0  0.0      0     0 ?        Z    07:50   0:00      \_ [sshd] <defunct>
1065534  32003  0.0  0.0      0     0 ?        Zs   07:50   0:00      \_ [cpu] <defunct>
1065534  32004  0.0  0.0  11452   724 ?        D    07:50   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534   2374  0.0  0.0      0     0 ?        Zs   07:51   0:00      \_ [cpu] <defunct>
1065534   2375  0.0  0.0  11452   724 ?        D    07:51   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534   5846  0.0  0.0      0     0 ?        Zs   07:52   0:00      \_ [forks] <defunct>
1065534   5847  0.0  0.0  22688  1632 ?        D    07:52   0:00      \_ awk /processes/ {print $2} /proc/stat
1000111   8955  0.0  0.0      0     0 ?        ZN   07:53   0:03      \_ [munin-html] <defunct>
1000111   8957  0.0  0.0      0     0 ?        ZN   07:53   0:09      \_ [munin-html] <defunct>
1000105  13289  0.0  0.0      0     0 ?        Z    07:54   0:00      \_ [sshd] <defunct>
1000105  15578  0.0  0.0      0     0 ?        Z    07:55   0:00      \_ [sshd] <defunct>
1000105  26119  0.0  0.0      0     0 ?        Z    07:59   0:00      \_ [sshd] <defunct>
1065534  28503  0.0  0.0      0     0 ?        Zs   08:00   0:00      \_ [cpu] <defunct>
1065534  28504  0.0  0.0  11452   724 ?        D    08:00   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105  29714  0.0  0.0      0     0 ?        Z    08:00   0:00      \_ [sshd] <defunct>
1000105  31098  0.0  0.0      0     0 ?        Z    08:00   0:00      \_ [sshd] <defunct>
1000105  31735  0.0  0.0      0     0 ?        Z    08:01   0:00      \_ [sshd] <defunct>
1065534  31761  0.0  0.0      0     0 ?        Zs   08:01   0:00      \_ [cpu] <defunct>
1065534  31762  0.0  0.0  11452   724 ?        D    08:01   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534   2412  0.0  0.0      0     0 ?        Zs   08:02   0:00      \_ [forks] <defunct>
1065534   2413  0.0  0.0  22688  1632 ?        D    08:02   0:00      \_ awk /processes/ {print $2} /proc/stat
1000111   6257  0.0  0.0      0     0 ?        ZN   08:03   0:03      \_ [munin-html] <defunct>
1000111   6258  0.0  0.0      0     0 ?        ZN   08:03   0:09      \_ [munin-html] <defunct>
1000105  13246  0.0  0.0      0     0 ?        Z    08:05   0:00      \_ [sshd] <defunct>
1000105  13248  0.0  0.0      0     0 ?        Z    08:05   0:00      \_ [sshd] <defunct>
1000105  20723  0.0  0.0      0     0 ?        Z    08:07   0:00      \_ [sshd] <defunct>
1000105  22329  0.0  0.0      0     0 ?        Z    08:08   0:00      \_ [sshd] <defunct>
1000105  28119  0.0  0.0      0     0 ?        Z    08:09   0:00      \_ [sshd] <defunct>
1065534  28717  0.0  0.0      0     0 ?        Zs   08:10   0:00      \_ [cpu] <defunct>
1065534  28725  0.0  0.0  11452   728 ?        D    08:10   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105  31866  0.0  0.0      0     0 ?        Z    08:11   0:00      \_ [sshd] <defunct>
1065534  31867  0.0  0.0      0     0 ?        Zs   08:11   0:00      \_ [cpu] <defunct>
1065534  31868  0.0  0.0  11452   724 ?        D    08:11   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105  32368  0.0  0.0      0     0 ?        Z    08:11   0:00      \_ [sshd] <defunct>
1000105    581  0.0  0.0      0     0 ?        Z    08:11   0:00      \_ [sshd] <defunct>
1000105   2737  0.0  0.0      0     0 ?        Z    08:12   0:00      \_ [sshd] <defunct>
1065534   3225  0.0  0.0      0     0 ?        Zs   08:12   0:00      \_ [forks] <defunct>
1065534   3226  0.0  0.0  22688  1632 ?        D    08:12   0:00      \_ awk /processes/ {print $2} /proc/stat
1000105   4934  0.0  0.0      0     0 ?        Z    08:12   0:00      \_ [sshd] <defunct>
1000111   7499  0.0  0.0      0     0 ?        ZN   08:13   0:03      \_ [munin-html] <defunct>
1000111   7500  0.0  0.0      0     0 ?        ZN   08:13   0:09      \_ [munin-html] <defunct>
1000105  10118  0.0  0.0      0     0 ?        Z    08:14   0:00      \_ [sshd] <defunct>
1000105  10356  0.0  0.0      0     0 ?        Z    08:14   0:00      \_ [sshd] <defunct>
1000105  10918  0.0  0.0      0     0 ?        Z    08:14   0:00      \_ [sshd] <defunct>
1000105  12143  0.0  0.0      0     0 ?        Z    08:14   0:00      \_ [sshd] <defunct>
1000105  18879  0.0  0.0      0     0 ?        Z    08:16   0:00      \_ [sshd] <defunct>
1000105  19841  0.0  0.0      0     0 ?        Z    08:16   0:00      \_ [sshd] <defunct>
1000105  20288  0.0  0.0      0     0 ?        Z    08:17   0:00      \_ [sshd] <defunct>
1000105  22005  0.0  0.0      0     0 ?        Z    08:17   0:00      \_ [sshd] <defunct>
1000105  22637  0.0  0.0      0     0 ?        Z    08:17   0:00      \_ [sshd] <defunct>
1000105  28337  0.0  0.0      0     0 ?        Z    08:19   0:00      \_ [sshd] <defunct>
1065534  29324  0.0  0.0      0     0 ?        Zs   08:20   0:00      \_ [cpu] <defunct>
1065534  29325  0.0  0.0  11452   724 ?        D    08:20   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105  30279  0.0  0.0      0     0 ?        Z    08:20   0:00      \_ [sshd] <defunct>
1065534  32460  0.0  0.0      0     0 ?        Zs   08:21   0:00      \_ [cpu] <defunct>
1065534  32461  0.0  0.0  11452   720 ?        D    08:21   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105    741  0.0  0.0      0     0 ?        Z    08:21   0:00      \_ [sshd] <defunct>
1065534   3512  0.0  0.0      0     0 ?        Zs   08:22   0:00      \_ [forks] <defunct>
1065534   3513  0.0  0.0  22688  1632 ?        D    08:22   0:00      \_ awk /processes/ {print $2} /proc/stat
1000111   7792  0.0  0.0      0     0 ?        ZN   08:23   0:03      \_ [munin-html] <defunct>
1000111   7793  0.0  0.0      0     0 ?        ZN   08:23   0:09      \_ [munin-html] <defunct>
1000105   9654  0.0  0.0      0     0 ?        Z    08:24   0:00      \_ [sshd] <defunct>
1000105  12691  0.0  0.0      0     0 ?        Z    08:24   0:00      \_ [sshd] <defunct>
1000105  19501  0.0  0.0      0     0 ?        Z    08:26   0:00      \_ [sshd] <defunct>
1000105  25305  0.0  0.0      0     0 ?        Z    08:28   0:00      \_ [sshd] <defunct>
1000105  28819  0.0  0.0      0     0 ?        Z    08:29   0:00      \_ [sshd] <defunct>
1065534  31704  0.0  0.0      0     0 ?        Zs   08:30   0:00      \_ [cpu] <defunct>
1065534  31706  0.0  0.0  11452   724 ?        D    08:30   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534   2654  0.0  0.0      0     0 ?        Zs   08:31   0:00      \_ [cpu] <defunct>
1065534   2655  0.0  0.0  11452   724 ?        D    08:31   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534   6355  0.0  0.0      0     0 ?        Zs   08:32   0:00      \_ [forks] <defunct>
1065534   6356  0.0  0.0  22688  1632 ?        D    08:32   0:00      \_ awk /processes/ {print $2} /proc/stat
1000105   9869  0.0  0.0      0     0 ?        Z    08:33   0:00      \_ [sshd] <defunct>
1000111   9884  0.0  0.0      0     0 ?        ZN   08:33   0:03      \_ [munin-html] <defunct>
1000111   9885  0.0  0.0      0     0 ?        ZN   08:33   0:09      \_ [munin-html] <defunct>
1000105  29011  0.0  0.0      0     0 ?        Z    08:38   0:00      \_ [sshd] <defunct>
1000105  29018  0.0  0.0      0     0 ?        Z    08:38   0:00      \_ [sshd] <defunct>
1000105  29028  0.0  0.0      0     0 ?        Z    08:38   0:00      \_ [sshd] <defunct>
1065534   1765  0.0  0.0      0     0 ?        Zs   08:40   0:00      \_ [cpu] <defunct>
1065534   1771  0.0  0.0  11452   724 ?        D    08:40   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105   2756  0.0  0.0      0     0 ?        Z    08:40   0:00      \_ [sshd] <defunct>
1065534   5491  0.0  0.0      0     0 ?        Zs   08:41   0:00      \_ [cpu] <defunct>
1065534   5492  0.0  0.0  11452   720 ?        D    08:41   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534   8674  0.0  0.0      0     0 ?        Zs   08:42   0:00      \_ [forks] <defunct>
1065534   8677  0.0  0.0  22688  1632 ?        D    08:42   0:00      \_ awk /processes/ {print $2} /proc/stat
1000105  10520  0.0  0.0      0     0 ?        Z    08:42   0:00      \_ [sshd] <defunct>
1000111  12896  0.0  0.0      0     0 ?        ZN   08:43   0:03      \_ [munin-html] <defunct>
1000111  12897  0.0  0.0      0     0 ?        ZN   08:43   0:09      \_ [munin-html] <defunct>
1000105  17089  0.0  0.0      0     0 ?        Z    08:44   0:00      \_ [sshd] <defunct>
1000105  17802  0.0  0.0      0     0 ?        Z    08:44   0:00      \_ [sshd] <defunct>
1000105  18439  0.0  0.0      0     0 ?        Z    08:45   0:00      \_ [sshd] <defunct>
1000105   2286  0.0  0.0      0     0 ?        Z    08:49   0:00      \_ [sshd] <defunct>
1065534   3213  0.0  0.0      0     0 ?        Zs   08:50   0:00      \_ [cpu] <defunct>
1065534   3218  0.0  0.0  11452   728 ?        D    08:50   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105   5252  0.0  0.0      0     0 ?        Z    08:50   0:00      \_ [sshd] <defunct>
1065534   6790  0.0  0.0      0     0 ?        Zs   08:51   0:00      \_ [cpu] <defunct>
1065534   6791  0.0  0.0  11452   724 ?        D    08:51   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  10132  0.0  0.0      0     0 ?        Zs   08:52   0:00      \_ [forks] <defunct>
1065534  10133  0.0  0.0  22688  1632 ?        D    08:52   0:00      \_ awk /processes/ {print $2} /proc/stat
1000111  14371  0.0  0.0      0     0 ?        ZN   08:53   0:03      \_ [munin-html] <defunct>
1000111  14372  0.0  0.0      0     0 ?        ZN   08:53   0:09      \_ [munin-html] <defunct>
1000105  19144  0.0  0.0      0     0 ?        Z    08:54   0:00      \_ [sshd] <defunct>
1000105  30477  0.0  0.0      0     0 ?        Z    08:57   0:00      \_ [sshd] <defunct>
1000105   6463  0.0  0.0      0     0 ?        Z    08:59   0:00      \_ [sshd] <defunct>
1000105   6781  0.0  0.0      0     0 ?        Z    08:59   0:00      \_ [sshd] <defunct>
1065534   7963  0.0  0.0      0     0 ?        Zs   09:00   0:00      \_ [cpu] <defunct>
1065534   7964  0.0  0.0  11452   724 ?        D    09:00   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105   8695  0.0  0.0      0     0 ?        Z    09:00   0:00      \_ [sshd] <defunct>
1065534  11294  0.0  0.0      0     0 ?        Zs   09:01   0:00      \_ [cpu] <defunct>
1065534  11297  0.0  0.0  11452   724 ?        D    09:01   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  15052  0.0  0.0      0     0 ?        Zs   09:02   0:00      \_ [forks] <defunct>
1065534  15053  0.0  0.0  22688  1632 ?        D    09:02   0:00      \_ awk /processes/ {print $2} /proc/stat
1000111  19310  0.0  0.0      0     0 ?        ZN   09:03   0:03      \_ [munin-html] <defunct>
1000111  19311  0.0  0.0      0     0 ?        ZN   09:03   0:09      \_ [munin-html] <defunct>
1000105  29646  0.0  0.0      0     0 ?        Z    09:06   0:00      \_ [sshd] <defunct>
1000105   6995  0.0  0.0      0     0 ?        Z    09:08   0:00      \_ [sshd] <defunct>
1000105   8024  0.0  0.0      0     0 ?        Z    09:08   0:00      \_ [sshd] <defunct>
1000105   9571  0.0  0.0      0     0 ?        Z    09:08   0:00      \_ [sshd] <defunct>
1065534  14943  0.0  0.0      0     0 ?        Zs   09:10   0:00      \_ [cpu] <defunct>
1065534  14944  0.0  0.0  11452   724 ?        D    09:10   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1000105  17651  0.0  0.0      0     0 ?        Z    09:10   0:00      \_ [sshd] <defunct>
1000105  18663  0.0  0.0      0     0 ?        Z    09:11   0:00      \_ [sshd] <defunct>
1065534  18706  0.0  0.0      0     0 ?        Zs   09:11   0:00      \_ [cpu] <defunct>
1065534  18707  0.0  0.0  11452   720 ?        D    09:11   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  22734  0.0  0.0      0     0 ?        Zs   09:12   0:00      \_ [forks] <defunct>
1065534  22735  0.0  0.0  22688  1632 ?        D    09:12   0:00      \_ awk /processes/ {print $2} /proc/stat
1000111  26848  0.0  0.0      0     0 ?        ZN   09:13   0:03      \_ [munin-html] <defunct>

So what is this strange thing supposed to be:

1065534  18707  0.0  0.0  11452   720 ?        D    09:11   0:00      \_ grep -E -q ^cpu +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ +[0-9]+ /proc/stat
1065534  22734  0.0  0.0      0     0 ?        Zs   09:12   0:00      \_ [forks] <defunct>
1065534  22735  0.0  0.0  22688  1632 ?        D    09:12   0:00      \_ awk /processes/ {print $2} /proc/stat

Out of nothing suddenly a jump to the last subuid with a strange grep command.

Also why are this zombi processes from subuid 1000105 grouped there suddenly in the middle of the tree of this

[lxc monitor] /var/snap/lxd/common/lxd/containers lxd313

?

I did not yet read myself through the deeps of how LXC works internally when assigning the fake UID’s to the processes. But especially this behaviour with subuid 1065534 looks strange to me.

This all happens again and again. So somehow this is triggered, but i was not able to identify where it comes from.

Any suggestions ?

This all runs on kernel: 3.10.0-1062.18.1.el7.x86_64 ( Centos 7 )
LXD Version 4.0.0 via snap

Thank you and stay healty ! :slight_smile:

Apart from the fact that we usually require kernel versions >= 3.13 I’d attribute this to a lxcfs bug we have had and fixed today. Once that rolls out the snap and you should restart your containers. If the issue reoccurs please file a bug or report here!