BlkIO missed in lxc-info output

nobody · November 20, 2023, 12:17pm

Hey there! After upgrade upto debian 12 and lxc 5, lxc-info always shows BlkIO use: 0 bytes

# lxc-info -n test-2.fr 
Name:           test-2.fr
State:          RUNNING
PID:            847679
IP:             192.168.51.208
CPU use:        0.88 seconds
BlkIO use:      0 bytes
Memory use:     49.13 MiB
KMem use:       5.06 MiB
Link:           vethiMj8Eq
 TX bytes:      1.65 KiB
 RX bytes:      706.34 KiB
 Total bytes:   707.99 KiB

Very strange thing, that the docker also shows 0 in BLOCK I/O docker stats. Maybe i should use some kernel/cgroups params or something?

# lxc-checkconfig                                                                                                                                                                     
LXC version 5.0.2                                                                                                                                                                                           
Kernel configuration not found at /proc/config.gz; searching...                                                                                                                                             
Kernel configuration found at /boot/config-6.1.0-10-amd64                                                                                                                                                   
                                                                                                                                                                                                            
--- Namespaces ---                                                                                                                                                                                          
Namespaces: enabled                                                                                                                                                                                         
Utsname namespace: enabled                                                                                                                                                                                  
Ipc namespace: enabled                                                                                                                                                                                      
Pid namespace: enabled                                                                                                                                                                                      
User namespace: enabled                                                                                                                                                                                     
Network namespace: enabled                                                                                                                                                                                  
                                                                                                                                                                                                            
--- Control groups ---                                                                                                                                                                                      
Cgroups: enabled                                                                                                                                                                                            
Cgroup namespace: enabled                                                                                                                                                                                   
Cgroup v1 mount points:                                                                                                                                                                                     
 - /sys/fs/cgroup/systemd
 - /sys/fs/cgroup/cpu,cpuacct
 - /sys/fs/cgroup/misc
 - /sys/fs/cgroup/net_cls,net_prio
 - /sys/fs/cgroup/devices
 - /sys/fs/cgroup/cpuset
 - /sys/fs/cgroup/memory
 - /sys/fs/cgroup/freezer
 - /sys/fs/cgroup/rdma
 - /sys/fs/cgroup/hugetlb
 - /sys/fs/cgroup/pids
 - /sys/fs/cgroup/blkio
 - /sys/fs/cgroup/perf_event
Cgroup v2 mount points: 
 - /sys/fs/cgroup/unified
Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, loaded

--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

stgraber · November 20, 2023, 3:43pm

That actually suggests that the relevant blkio files are actually present (io.stat mostly) but that the file doesn’t contain any useful metrics for the container.

I/O tracking and limiting has been quite problematic for a few years as most of the I/O scheduling logics these days just shove everything at the drive and let it deal with it (noop scheduler used for most SCSI and NVME devices).

nobody · November 21, 2023, 9:07am

Hey, Stephane! Glad to see you!
I have no idea about io.stat but on debian 12 there are many IO pseudo-files just missed. Difference between debian 10 and 12 below:

root@debian10:~# ll /sys/fs/cgroup/blkio/ns/test-1.fr/
total 0
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.io_merged
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.io_merged_recursive
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.io_queued
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.io_queued_recursive
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.io_service_bytes
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.io_service_bytes_recursive
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.io_serviced
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.io_serviced_recursive
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.io_service_time
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.io_service_time_recursive
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.io_wait_time
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.io_wait_time_recursive
-rw-r--r-- 1 root root 0 Jul 26 16:53 blkio.leaf_weight
-rw-r--r-- 1 root root 0 Jul 26 16:53 blkio.leaf_weight_device
--w------- 1 root root 0 Jul 26 16:53 blkio.reset_stats
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.sectors
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.sectors_recursive
-r--r--r-- 1 root root 0 Aug 26  2022 blkio.throttle.io_service_bytes
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.throttle.io_service_bytes_recursive
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.throttle.io_serviced
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.throttle.io_serviced_recursive
-rw-r--r-- 1 root root 0 Jul 26 16:53 blkio.throttle.read_bps_device
-rw-r--r-- 1 root root 0 Jul 26 16:53 blkio.throttle.read_iops_device
-rw-r--r-- 1 root root 0 Jul 26 16:53 blkio.throttle.write_bps_device
-rw-r--r-- 1 root root 0 Jul 26 16:53 blkio.throttle.write_iops_device
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.time
-r--r--r-- 1 root root 0 Jul 26 16:53 blkio.time_recursive
-rw-r--r-- 1 root root 0 Jul 26 16:53 blkio.weight
-rw-r--r-- 1 root root 0 Jul 26 16:53 blkio.weight_device
-rw-r--r-- 1 root root 0 Jul 26 16:53 cgroup.clone_children
-rw-r--r-- 1 root root 0 Nov  7 12:47 cgroup.procs
-rw-r--r-- 1 root root 0 Jul 26 16:53 notify_on_release
-rw-r--r-- 1 root root 0 Jul 26 16:53 tasks

root@debian12:~# ll /sys/fs/cgroup/blkio/lxc.monitor.test-2.fr/
total 0
--w------- 1 root root 0 Nov 20 22:18 blkio.reset_stats
-r--r--r-- 1 root root 0 Nov 20 22:18 blkio.throttle.io_service_bytes
-r--r--r-- 1 root root 0 Nov 20 22:18 blkio.throttle.io_service_bytes_recursive
-r--r--r-- 1 root root 0 Nov 20 22:18 blkio.throttle.io_serviced
-r--r--r-- 1 root root 0 Nov 20 22:18 blkio.throttle.io_serviced_recursive
-rw-r--r-- 1 root root 0 Nov 20 22:18 blkio.throttle.read_bps_device
-rw-r--r-- 1 root root 0 Nov 20 22:18 blkio.throttle.read_iops_device
-rw-r--r-- 1 root root 0 Nov 20 22:18 blkio.throttle.write_bps_device
-rw-r--r-- 1 root root 0 Nov 20 22:18 blkio.throttle.write_iops_device
-rw-r--r-- 1 root root 0 Nov 20 22:18 cgroup.clone_children
-rw-r--r-- 1 root root 0 Nov 20 22:18 cgroup.procs
-rw-r--r-- 1 root root 0 Nov 20 22:18 notify_on_release
-rw-r--r-- 1 root root 0 Nov 20 22:18 tasks

And the same we can see on /sys/fs/cgroup/blkio/docker/
Well, i still think i missed some kernel param…

stgraber · November 21, 2023, 4:27pm

That’s odd, I thought Debian 12 had moved to cgroup2 by default.

nobody · November 22, 2023, 9:01pm

It had, but i use systemd.unified_cgroup_hierarchy=false to get a hybrid cgroups . Maybe this is the reason? Dunno but it’s hardly likely.