Systemd-tmpfiles-clean.service - Failed to start Cleanup of Temporary Directories


(Priit3344) #1

Hi,

Im having an issues with some of my containers. In time it seems systemd-tmpfiles-clean.service ends up failing.

systemd-tmpfiles-clean.service: main process exited, code=exited, status=219/CGROUP
systemd[1]: Failed to start Cleanup of Temporary Directories.
Failed to kill control group: Transport endpoint is not connected
Failed to create cgroup /lxc/container_name/system.slice/systemd-initctl.service: Transport endpoint is not connected

df -h gives also,

df: ‘/proc/cpuinfo’: Transport endpoint is not connected
df: ‘/proc/diskstats’: Transport endpoint is not connected
df: ‘/proc/meminfo’: Transport endpoint is not connected
df: ‘/proc/stat’: Transport endpoint is not connected
df: ‘/proc/swaps’: Transport endpoint is not connected
df: ‘/proc/uptime’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/blkio’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/cpuacct,cpu’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/cpuset’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/devices’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/freezer’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/hugetlb’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/memory’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/systemd’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/net_prio,net_cls’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/perf_event’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/pids’: Transport endpoint is not connected

Host.
3.10.0-862.11.6.el7.x86_64 #1 SMP Tue Aug 14 21:49:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
LXC container
3.10.0-862.11.6.el7.x86_64 #1 SMP Tue Aug 14 21:49:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

The only thing what seems to work at the moment is when I,

systemctl --force reboot

otherwise it wont reboot or halt -p is not working.

Is there any fix to this issue or anyone has encountered the same issue ?


(Stéphane Graber) #2

Output above suggests that you’re using lxcfs and that lxcfs on the host has crashed or has been manually restarted which would cause this kind of behavior in any running container.

Note that lxcfs updates should never actually restart it, instead SIGUSR1 should be sent to it so that it can safely re-exec itself, keeping all containers happy.


(Priit3344) #3

The thing is I dont see any lxcfs service or any kind of trace of it. Can it be something else that crashed those endpoints ?

systemctl status snap.lxd.daemon.service -l

● snap.lxd.daemon.service - Service for snap application lxd.daemon
Loaded: loaded (/etc/systemd/system/snap.lxd.daemon.service; static; vendor preset: disabled)
Active: active (running) since Thu 2019-01-10 23:01:27 EET; 5 days ago
Main PID: 26374 (daemon.start)
CGroup: /system.slice/snap.lxd.daemon.service
‣ 26374 /bin/sh /snap/lxd/9886/commands/daemon.start

Jan 13 01:01:01 lxd.daemon[26374]: bindings.c: 823: cgfs_iterate_cgroup: Failed closedir for lxc.payload/container1/user.slice/user-0.slice/session-c53.scope: Bad file descriptor
Jan 13 01:01:01 lxd.daemon[26374]: bindings.c: 823: cgfs_iterate_cgroup: Failed closedir for lxc.payload/container2/user.slice/user-0.slice/session-c54.scope: Bad file descriptor
Jan 13 01:01:01 lxd.daemon[26374]: bindings.c: 626: recursive_rmdir: Failed to close directory lxc.payload/container3/user.slice/user-0.slice/session-c53.scope: Bad file descriptor
Jan 13 02:01:01 lxd.daemon[26374]: bindings.c: 823: cgfs_iterate_cgroup: Failed closedir for lxc.payload/container4/user.slice/user-0.slice/session-c54.scope: Bad file descriptor
Jan 13 02:01:01 lxd.daemon[26374]: bindings.c: 823: cgfs_iterate_cgroup: Failed closedir for lxc.payload/container5/user.slice/user-0.slice/session-c54.scope: Bad file descriptor
Jan 15 13:51:17 lxd.daemon[26374]: t=2019-01-15T13:51:17+0200 lvl=eror msg=“Failed to remove disk device path” err=“remove /var/snap/lxd/common/lxd/devices/loan-application-backend-lv-dev/disk.loan-application-backend-lv-dev.var-lib-pgsql-10: no such file or directory” path=/var/snap/lxd/common/lxd/devices/loan-application-backend-lv-dev/disk.loan-application-backend-lv-dev.var-lib-pgsql-10
Jan 15 14:28:38 lxd.daemon[26374]: t=2019-01-15T14:28:38+0200 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Jan 15 17:46:40 lxd.daemon[26374]: t=2019-01-15T17:46:40+0200 lvl=warn msg=“Detected poll(POLLNVAL) event: exiting.”
Jan 15 17:46:40 lxd.daemon[26374]: t=2019-01-15T17:46:40+0200 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Jan 16 09:26:13 lxd.daemon[26374]: t=2019-01-16T09:26:13+0200 lvl=warn msg=“Detected poll(POLLNVAL) event.”

df: ‘/proc/cpuinfo’: Transport endpoint is not connected
df: ‘/proc/diskstats’: Transport endpoint is not connected
df: ‘/proc/meminfo’: Transport endpoint is not connected
df: ‘/proc/stat’: Transport endpoint is not connected
df: ‘/proc/swaps’: Transport endpoint is not connected
df: ‘/proc/uptime’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/blkio’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/cpuacct,cpu’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/cpuset’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/devices’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/freezer’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/hugetlb’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/memory’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/systemd’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/net_prio,net_cls’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/perf_event’: Transport endpoint is not connected
df: ‘/sys/fs/cgroup/pids’: Transport endpoint is not connected