Hi,
i have more than 50 containers on a hostnode and almost all of them returning “Error: /proc must be mounted” error when I run free -m ,top, htop command. Restarting the container fix the issue.
LXD version =
server_version: “4.9”
storage: zfs
storage_version: 0.7.12-2+deb10u2
I suspect the lxcfs service was stopped for some reason.
@tomp ps fauxww | grep lxcfs output is given below.
root 2544 0.0 0.0 150680 0 ? Ssl Feb09 0:00 /usr/bin/lxcfs /var/lib/lxcfs/
root 26988 0.0 0.0 4836 836 pts/3 S+ 11:28 0:00 | _ grep lxcfs
root 3509 0.0 0.0 828052 4536 ? Sl Feb09 0:58 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
Once lxcfs is restarted, any previously started containers will need to be restarted as their link to lxcfs will be severed.
how to avoid lxcfs from crashing or stopping ???.
@brauner is there a way to “reconnect” a running container to the lxcfs mounts after it has died/restarted?
No, only option is to unmount the lxcfs mounts if you want the container to keep working, then restart the container when convenient.
@tomp so the only best possible way to fix it is rebooting container at suitable time due to technical reasons, but is there a way to avoid it ??? since my automation and monitoring scripts depend upon free -m and top commands etc.
Look for a crash message from LXCFS in journalctl -u snap.lxd.daemon, if you find one, then maybe we can figure out what the bug is.
– Logs begin at Tue 2021-02-23 19:07:01 UTC, end at Thu 2021-03-04 15:01:56 UTC. –
Feb 24 00:04:30 lxd.daemon[3210]: 2021/02/24 00:04:30 http: TLS handshake error from :36806: tls: client offered only unsuppor
Feb 24 00:04:41 lxd.daemon[3210]: 2021/02/24 00:04:41 http: TLS handshake error from :34726: tls: no cipher suite supported by
Feb 24 00:05:08 lxd.daemon[3210]: 2021/02/24 00:05:08 http: TLS handshake error from :33268: EOF
Feb 24 00:05:09 lxd.daemon[3210]: 2021/02/24 00:05:09 http: TLS handshake error from :39642: EOF
Feb 24 00:05:09 lxd.daemon[3210]: 2021/02/24 00:05:09 http: TLS handshake error from :45788: EOF
Feb 24 00:05:09 lxd.daemon[3210]: 2021/02/24 00:05:09 http: TLS handshake error from :52052: tls: no cipher suite supported by
Feb 24 00:05:10 lxd.daemon[3210]: 2021/02/24 00:05:10 http: TLS handshake error from :52298: EOF
Feb 24 00:05:10 lxd.daemon[3210]: 2021/02/24 00:05:10 http: TLS handshake error from :57826: tls: client offered only unsuppor
Feb 24 00:05:10 lxd.daemon[3210]: 2021/02/24 00:05:10 http: TLS handshake error from :58044: EOF
Feb 24 00:05:11 lxd.daemon[3210]: 2021/02/24 00:05:11 http: TLS handshake error from :34324: EOF
Feb 24 00:05:11 lxd.daemon[3210]: 2021/02/24 00:05:11 http: TLS handshake error from :38072: EOF
Feb 24 00:05:12 lxd.daemon[3210]: 2021/02/24 00:05:12 http: TLS handshake error from :41056: EOF
Feb 24 00:45:27 lxd.daemon[3210]: 2021/02/24 00:45:27 http: TLS handshake error from :48734: read tcp 10.168.0.5:8443->162.142.
Feb 24 02:45:16 lxd.daemon[3210]: 2021/02/24 02:45:16 http: TLS handshake error from :61000: tls: client offered only unsuppor
Feb 24 03:38:30 lxd.daemon[3210]: 2021/02/24 03:38:30 http: TLS handshake error from :44640: EOF
Feb 24 03:38:31 lxd.daemon[3210]: 2021/02/24 03:38:31 http: TLS handshake error from :46712: EOF
Feb 24 03:38:31 lxd.daemon[3210]: 2021/02/24 03:38:31 http: TLS handshake error from :48619: EOF
Feb 24 03:38:31 lxd.daemon[3210]: 2021/02/24 03:38:31 http: TLS handshake error from :50250: tls: no cipher suite supported by b
Feb 24 03:38:32 lxd.daemon[3210]: 2021/02/24 03:38:32 http: TLS handshake error from :52120: EOF
Feb 24 03:38:32 lxd.daemon[3210]: 2021/02/24 03:38:32 http: TLS handshake error from :21772: tls: client offered only unsuppor
Feb 24 03:38:33 lxd.daemon[3210]: 2021/02/24 03:38:33 http: TLS handshake error from :5070: EOF
Feb 24 03:38:33 lxd.daemon[3210]: 2021/02/24 03:38:33 http: TLS handshake error from :33006: EOF
Feb 24 03:38:34 lxd.daemon[3210]: 2021/02/24 03:38:34 http: TLS handshake error from :29056: EOF
Feb 24 03:38:34 lxd.daemon[3210]: 2021/02/24 03:38:34 http: TLS handshake error from :27434: EOF
Feb 24 08:00:09 lxd.daemon[3210]: t=2021-02-24T08:00:09+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Feb 24 08:00:10 lxd.daemon[3210]: t=2021-02-24T08:00:10+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Feb 24 08:00:10 lxd.daemon[3210]: t=2021-02-24T08:00:10+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Feb 24 08:00:11 lxd.daemon[3210]: t=2021-02-24T08:00:11+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Feb 24 08:00:11 lxd.daemon[3210]: t=2021-02-24T08:00:11+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Feb 24 08:00:12 lxd.daemon[3210]: t=2021-02-24T08:00:12+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Feb 24 08:00:12 lxd.daemon[3210]: t=2021-02-24T08:00:12+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Feb 24 08:00:13 lxd.daemon[3210]: t=2021-02-24T08:00:13+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Feb 24 08:00:13 lxd.daemon[3210]: t=2021-02-24T08:00:13+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Feb 24 08:00:14 lxd.daemon[3210]: t=2021-02-24T08:00:14+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Feb 24 08:00:15 lxd.daemon[3210]: t=2021-02-24T08:00:15+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Feb 24 08:00:16 lxd.daemon[3210]: t=2021-02-24T08:00:16+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Feb 24 08:00:16 lxd.daemon[3210]: t=2021-02-24T08:00:16+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Feb 24 08:00:17 lxd.daemon[3210]: t=2021-02-24T08:00:17+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Feb 24 08:00:17 lxd.daemon[3210]: t=2021-02-24T08:00:17+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Feb 24 08:00:18 lxd.daemon[3210]: t=2021-02-24T08:00:18+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
Feb 24 08:00:18 lxd.daemon[3210]: t=2021-02-24T08:00:18+0000 lvl=warn msg=“Detected poll(POLLNVAL) event.”
no crash message in the command you provided.
yeah, no LXCFS crash in there
Not really, the fuse connection is broken so this would require a
container restart unfortunately.
Simply unmount all lxcfs mounts in the container will get back into
working (albeit without proc-based value virtualization) state.
Christian
1 Like