Lxcfs spamming syslog with logs "lxc.payload.container has unexpected cpu time; unable to determine idle time"

saleem · May 28, 2021, 2:15pm

Recently, I upgraded my ubuntu host from 16.04 to 20.04 version. After the host upgrade, the upgraded lxc version is 4.0.2 and lxcfs is 4.0.3.

In the upgraded host, the host started to receive lxcfs syslog errors like this for every few seconds for all the containers

May 18 22:25:00 us-host4 lxcfs[2201]: proc_fuse.c: 876: proc_stat_read: cpuX from /lxc.payload.app1 has unexpected cpu time: 23457536 in /proc/stat, 23484514 in cpuacct.usage_all; unable to determine idle time
May 18 22:25:00 us-host4 lxcfs[2201]: proc_fuse.c: 876: proc_stat_read: cpuX from /lxc.payload.app1 has unexpected cpu time: 23409824 in /proc/stat, 23484757 in cpuacct.usage_all; unable to determine idle time
May 18 22:25:00 us-host4 lxcfs[2201]: proc_fuse.c: 876: proc_stat_read: cpuX from /lxc.payload.app1 has unexpected cpu time: 23427791 in /proc/stat, 23512092 in cpuacct.usage_all; unable to determine idle time
May 18 22:25:00 us-host4 lxcfs[2201]: proc_fuse.c: 876: proc_stat_read: cpuX from /lxc.payload.app1 has unexpected cpu time: 23410695 in /proc/stat, 23527624 in cpuacct.usage_all; unable to determine idle time
May 18 22:25:00 us-host4 lxcfs[2201]: proc_fuse.c: 876: proc_stat_read: cpuX from /lxc.payload.app1 has unexpected cpu time: 23417370 in /proc/stat, 23546309 in cpuacct.usage_all; unable to determine idle time

Version:

OS: Ubuntu 20.04.2 LTS
Kernel: 5.4.0-72-generic
lxc-ls: 4.0.2
lxcfs: 4.0.3
Containers OS: Ubuntu 20.04.2 LTS

I have tried rebooting the host as mentioned in the comments on other related issues. It didn’t work.

Manually I build and upgraded lxcfs version to 4.0.8. But still these logs are showing up.

Related issues reported already but I don’t find any solutions there.

I have posted this already on github lxcfs spamming syslog with logs "lxc.payload.container has unexpected cpu time; unable to determine idle time" · Issue #464 · lxc/lxcfs · GitHub

Is this a bug or something configured wrongly?

stgraber · May 28, 2021, 3:50pm

@brauner

brauner · May 31, 2021, 2:19pm

I think we should dumb down the logging tbh as it’s not really useful.

brauner · May 31, 2021, 2:29pm

Why does the log output show cpuX, i.e. why X and not a specific cpu number this doesn’t match with the code I’m looking at.

saleem · May 31, 2021, 3:00pm

@brauner I just edited that particular text cpuX . Sorry for that.
Here is the logs with actual CPU numbers

May 18 22:25:00 us-host4 lxcfs[2201]: proc_fuse.c: 876: proc_stat_read: cpu33 from /lxc.payload.app1 has unexpected cpu time: 23457536 in /proc/stat, 23484514 in cpuacct.usage_all; unable to determine idle time
May 18 22:25:00 us-host4 lxcfs[2201]: proc_fuse.c: 876: proc_stat_read: cpu34 from /lxc.payload.app1 has unexpected cpu time: 23409824 in /proc/stat, 23484757 in cpuacct.usage_all; unable to determine idle time
May 18 22:25:00 us-host4 lxcfs[2201]: proc_fuse.c: 876: proc_stat_read: cpu35 from /lxc.payload.app1 has unexpected cpu time: 23427791 in /proc/stat, 23512092 in cpuacct.usage_all; unable to determine idle time
May 18 22:25:00 us-host4 lxcfs[2201]: proc_fuse.c: 876: proc_stat_read: cpu36 from /lxc.payload.app1 has unexpected cpu time: 23410695 in /proc/stat, 23527624 in cpuacct.usage_all; unable to determine idle time
May 18 22:25:00 us-host4 lxcfs[2201]: proc_fuse.c: 876: proc_stat_read: cpu37 from /lxc.payload.app1 has unexpected cpu time: 23417370 in /proc/stat, 23546309 in cpuacct.usage_all; unable to determine idle time

brauner · May 31, 2021, 3:07pm

Ah, ok. It’s just for me to know whether or not I’m dealing with a custom patched LXCFS version or not.

saleem · June 1, 2021, 3:10pm

@brauner Could you explain why this logs came? Is there something wrong in the container?

stgraber · June 1, 2021, 3:27pm

It can happen on very busy containers or those hitting lxcfs particularly hard.
The kernel API for this is inherently racy and when such a race is hit, lxcfs logs it.

brauner · June 1, 2021, 4:37pm

And we should’ve logged it at debug level the whole time and not error level.

saleem · June 6, 2021, 1:19pm

Got it. Many thanks for your responses @stgraber @brauner

saleem · June 8, 2021, 8:00am

@stgraber @brauner FYI, these logs are coming only in the upgraded ubuntu lxc host from 16.04 to 20.04 or from 18.04 to 20.04. In the fresh install of Ubuntu 20.04, using lxcfs version 4.0.3 doesn’t give these errors.