Best way to monitor per-container I/O (iostat) statistics in Incus (ZFS directory-backed storage)?

Incus
1

Hi

Really loving incus! I’m starting to have some learning/growing pains as I expand usage of it.

I noticed some io hangings that I traced to one of the containers (it is running dedicated gitlab) but finding which container was causing the io bottleneck was an involved process. I’m wondering if there is a newer or native way to generate per-container I/O stats?

I saw a previous discussion about this on the forum here: LXD - Per-container stats , but it’s a few years old and I’m hoping there is updated info or features?

The setup:

  • Host OS: Debian 13 (Trixie)
  • Incus server using a ZFS pool (default) mirrored across two physical SSDs (/dev/sdc and /dev/sdd).
  • Container Storage: Both Standard filesystem-backed directories/datasets and VMs using dedicated block volumes. (The one causing issues was in the filesystem directory)

Things I’ve tried:

  • iostat -x 1 1 : shows the aggregate raw /dev/sdX metrics but can’t isolate individual container filesystem datasets (but does for block drives)
  • incus top : lacks any disk I/O metrics.
  • iotop: can use but have to scrape output (see below)

If I run iotop -b -a -o -n 12 -d 5 > /tmp/iotop_summary.txt I’ll get a report which lists USER (uid) and TID like so

TID     PRIO USER        DISK READ  DISK WRITE  SWAPIN IO    COMMAND
4130527 be/4 1000000       0.00 B      2.57 M ?unavailable?  REDACTED
2170159 be/4 1000000       0.00 B    538.00 K ?unavailable?  REDACTED
4168300 be/4 1000987       0.00 B      4.00 K ?unavailable?  REDACTED
...

Because my unprivileged containers share the same namespace range, the USER column overlapping prevents mapping them directly. Right now, my workaround is parsing the individual Thread ID back to its cgroup v2 lxc payload to get which container it belongs to:

E.g.

$ cat /proc/4168300/cgroup
0::/lxc.payload.CONTAINER1/REDACTED.slice/REDACTED.service
$ cat /proc/4130527/cgroup
0::/lxc.payload.CONTAINER2/REDACTED.slice/REDACTED.service

so I could work it out and create a script to automate all that.

Is there a built-in Incus command, a ZFS property/command, or a recommended best practice to get a clean, aggregated iostat or iotop style dashboard per filesystem-backed container?

Thanks in advance!

P.S. I solved the container hanging issue with zfs set sync=disabled default/containers/CONTAINER1 (and a few other zfs tuning things). Yes I know that’s not the safest, but ok with that application/container.

2

Would the metrics endpoint work for your needs?

3

I don’t think so. When I look at the metrics it tracks aggregate data transfer, not the await io data you get with iostat and (I think) it mixes up the drive info in the container with the drive info on the host

For example when I run incus query of metrics I get

 $ incus query /1.0/metrics | grep disk | grep CONATINER1
incus_disk_read_bytes_total{device="zd32",name="CONTAINER1",project="default",type="container"} 0
incus_disk_read_bytes_total{device="sda",name="CONTAINER1",project="default",type="container"} 0
incus_disk_read_bytes_total{device="sdb",name="CONTAINER1",project="default",type="container"} 0
incus_disk_read_bytes_total{device="md1",name="CONTAINER1",project="default",type="container"} 0
incus_disk_reads_completed_total{device="zd32",name="CONTAINER1",project="default",type="container"} 0
incus_disk_reads_completed_total{device="sda",name="CONTAINER1",project="default",type="container"} 0
incus_disk_reads_completed_total{device="sdb",name="CONTAINER1",project="default",type="container"} 0
incus_disk_reads_completed_total{device="md1",name="CONTAINER1",project="default",type="container"} 0
incus_disk_written_bytes_total{device="zd32",name="CONTAINER1",project="default",type="container"} 0
incus_disk_written_bytes_total{device="sda",name="CONTAINER1",project="default",type="container"} 122880
incus_disk_written_bytes_total{device="sdb",name="CONTAINER1",project="default",type="container"} 122880
incus_disk_written_bytes_total{device="md1",name="CONTAINER1",project="default",type="container"} 114688
incus_disk_writes_completed_total{device="zd32",name="CONTAINER1",project="default",type="container"} 0
incus_disk_writes_completed_total{device="sda",name="CONTAINER1",project="default",type="container"} 30
incus_disk_writes_completed_total{device="sdb",name="CONTAINER1",project="default",type="container"} 30
incus_disk_writes_completed_total{device="md1",name="CONTAINER1",project="default",type="container"} 30

so I see total written #s (very low), and the wrong drives (sda, sdb, md1, zd32) despite the fact that CONTAINER1 lives entirely under default in the zpool of sdc/sdd

I couldn’t find anything there like iostat latency data.

4

Because my unprivileged containers share the same namespace range

-- change the namespace range for each container then, very easy to deploy and generally safer as well.