Systemd-cgtop has nothing in the Input/Output columns

1

Hello,

I’m sorry if this question is not about LXC itself but I couldn’t find the solution anywhere else.

I’m trying to monitor resource usage of LXC containers by running systemd-cgtop on Ubuntu 20.04 host.
CPU and Memory columns are fine but both Input and Output columns always have “-” in them even under a heavy disk load.

I tried changing these systemd settings but it didn’t help (tried both yes and no):

DefaultIOAccounting=yes
DefaultBlockIOAccounting=yes

I was testing disk usage with dd if=/dev/sda of=/dev/null. Here’s the output of systemd-cgtop:

Control Group     Tasks   %CPU   Memory  Input/s Output/s
/                   208  105.4     3.7G        -        -
user.slice           16  101.1     3.1G        -        -
system.slice         87    0.0   391.9M        -        -

Any help would be appreciated.

2

Block accounting on Linux has been pretty useless for a little while.
That’s because only a single I/O scheduler (CFQ) ever implemented it properly and most Linux distributions have moved away from it.

These days, most systems use SCSI multi-queue (mq) and NVME drives, which uses the noop scheduler. As the name implies, that scheduler does nothing and just queues all requests straight in hardware. This effectively prevents us from pulling any useful statistics or apply any I/O limits.

3

Thank you for the answer but it seems the problem is that Ubuntu 20.04 doesn’t use cgroups v2 hierarchy:

$ systemctl --version
systemd 245 (245.4-4ubuntu3.19)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

I added systemd.unified_cgroup_hierarchy=1 to the kernel’s command line and now systemd-cgtop shows Input/Output. But I ran into another problem, my containers didn’t start:

$ lxc-start ansible -F
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...

I did some searching and found the solution:

lxc.init.cmd = /sbin/init systemd.unified_cgroup_hierarchy=1

However, I’m not sure my experiments make sense and won’t break anything else in the future. Should I just stick to a default cgroups hierarchy?

4

Ubuntu 20.04 is indeed a cgroup1 distro by default. Ubuntu 22.04 comes with cgroup2 instead.

Your container failing to start out of the box is most likely because it’s running one of those distros that won’t handle cgroup2 properly out of the box. We mostly see this on CentOS 7 or Ubuntu 16.04, newer distros tend to just go with whatever is enabled on the system.