Container CPU shows high sy/us in /proc/stat (Zabbix alert) while its cgroup shows <1% usage. Pinned cores shared with a noisy neighbor — recommended isolation and monitoring setup?

Incus
1

Hi all,

We are running a multi-tenant Incus host and observed a CPU monitoring discrepancy. We would like to confirm the behavior and recommended setup.

Environment

  • Host: 96 cores, ~1 TB RAM
  • Kernel: 6.8.0-86-generic
  • Incus: 6.18
  • lxcfs: bundled with Incus; standalone lxcfs --version is not available
  • Ubuntu 24.04 containers, cgroup v2
  • lxcfs:
    text
    /opt/incus/bin/lxcfs /var/lib/incus-lxcfs --enable-loadavg

Affected container:

yaml
limits.cpu: “4”
limits.memory: 4GiB

Its effective cpuset is:

text
83,86,91,94

However, some other containers also have overlapping CPUs. One customer container (tenant-a) has no CPU limit and runs Docker workloads.

Observed behavior

Zabbix runs inside the container and reads /proc/stat. It reports around 37% CPU.

Inside the container:

text
top:
7.8 us, 28.7 sy, 63.5 id

load average: 0.01

But the container’s cgroup shows almost no CPU usage:

text
usage_usec delta over 10s: 268,298 us
system_usec delta: 38,917 us
cpu.pressure: 0.00
nr_throttled: 0

On the host, the four CPUs assigned to the container show:

text
CPU 83: 14.3 usr, 61.2 sys, 24.6 idle
CPU 86: 2.0 usr, 0.9 sys, 97.1 idle
CPU 91: 14.6 usr, 59.5 sys, 25.9 idle
CPU 94: 2.4 usr, 0.9 sys, 97.6 idle

pidstat showed that the busy CPUs were primarily running processes from the unlimited tenant-a container.

Our understanding

Our current understanding is:

  1. limits.cpu: N assigns CPUs through a cpuset but does not exclusively reserve those CPUs.
  2. Other containers can therefore run on the same CPUs.
  3. /proc/stat inside the container may reflect activity on those CPUs, causing Zabbix/top to show CPU usage generated by neighboring containers.
  4. The container’s cgroup metrics (cpu.stat, cpu.pressure) represent its actual CPU usage more accurately.

Questions

  1. Is this /proc/stat behavior expected with lxcfs/Incus 6.18?
  2. Can lxcfs or Incus make /proc/stat/top show only the container’s own CPU usage?
  3. Would --enable-cfs or --enable-cpuview help, particularly with cpuset-based CPU allocation?
  4. For Zabbix, should we monitor cgroup cpu.stat/cpu.pressure or Incus host-side metrics instead of /proc/stat?
  5. For multi-tenant workloads, would you recommend combining:
    • non-overlapping cpusets for critical workloads,
    • limits.cpu.allowance,
    • limits.cpu.priority,
    • limits.processes,
    • I/O limits?
  6. Does Incus dynamically choose/rebalance CPUs when using limits.cpu: N, or is the CPU assignment static?
  7. Is it recommended to reserve some CPUs for host/Incus services, and if so, what is the preferred cgroup/systemd approach?

Thanks!

2

@stgraber Your assitance is required. Thanks!

3
  1. I don’t know, that’s a LXCFS thing and that can vary widely based on kernel version
  2. /proc/stat is implemented by LXCFS
  3. Possibly
  4. Don’t run monitoring agents in containers, Incus has a /1.0/metrics with more reliable metrics instead. A monitoring agent in a container is likely to cause more overall load than what’s running in the container.
  5. Don’t run multi-tenant environments using containers, there are far too many user to root privilege escalation exploits around for that to be safe. You want VMs to separate tenants.
  6. It does re-balance every time a container starts or stops on the systems (or limits.cpu is change)
  7. It’s possible but I wouldn’t call that recommended. I believe we consider the parent cgroup when doing placement. Otherwise we isolcpus as a kernel option + direct pining of the control plane services also works.