IOPS Limits with ZFS storage backend doesn't seem to work

Philip_C · February 10, 2023, 9:40am

While setting the IOPS to a container running with the DIR backend, the IOPS restrictions seem to work.

Without IOPS config (DIR backend)


fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 145.56 MB/s  (36.3k) | 1.41 GB/s    (22.1k)
Write      | 145.94 MB/s  (36.4k) | 1.42 GB/s    (22.2k)
Total      | 291.51 MB/s  (72.8k) | 2.84 GB/s    (44.4k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 2.19 GB/s     (4.2k) | 2.24 GB/s     (2.1k)
Write      | 2.31 GB/s     (4.5k) | 2.39 GB/s     (2.3k)
Total      | 4.51 GB/s     (8.8k) | 4.64 GB/s     (4.5k)


DIR backend (with r/w IOPS set to 100)

fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 378.00 KB/s     (94) | 6.04 MB/s       (94)
Write      | 400.00 KB/s    (100) | 6.40 MB/s      (100)
Total      | 778.00 KB/s    (194) | 12.44 MB/s     (194)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 48.46 MB/s      (94) | 69.72 MB/s      (68)
Write      | 51.32 MB/s     (100) | 74.36 MB/s      (72)
Total      | 99.78 MB/s     (194) | 144.08 MB/s    (140)



DIR backend (with r/w IOPS set to 300)



fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 1.16 MB/s      (291) | 18.62 MB/s     (291)
Write      | 1.20 MB/s      (300) | 19.20 MB/s     (300)
Total      | 2.36 MB/s      (591) | 37.83 MB/s     (591)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 145.86 MB/s    (284) | 161.23 MB/s    (157)
Write      | 153.61 MB/s    (300) | 171.96 MB/s    (167)
Total      | 299.48 MB/s    (584) | 333.19 MB/s    (324)

As you can see, the results are satisfactory.

However, with ZFS as the backend the IOPS are very much identical no matter how low the IOPS limits are set to.

Without IOPS limit (ZFS backend)

fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 95.29 MB/s   (23.8k) | 448.65 MB/s   (7.0k)
Write      | 95.54 MB/s   (23.8k) | 451.02 MB/s   (7.0k)
Total      | 190.83 MB/s  (47.7k) | 899.67 MB/s  (14.0k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 2.50 GB/s     (4.9k) | 2.49 GB/s     (2.4k)
Write      | 2.64 GB/s     (5.1k) | 2.65 GB/s     (2.5k)
Total      | 5.15 GB/s    (10.0k) | 5.14 GB/s     (5.0k)


ZFS backend r/w IOPS: 50/50

fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 100.25 MB/s  (25.0k) | 630.58 MB/s   (9.8k)
Write      | 100.52 MB/s  (25.1k) | 633.90 MB/s   (9.9k)
Total      | 200.77 MB/s  (50.1k) | 1.26 GB/s    (19.7k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 2.64 GB/s     (5.1k) | 2.39 GB/s     (2.3k)
Write      | 2.78 GB/s     (5.4k) | 2.55 GB/s     (2.4k)
Total      | 5.42 GB/s    (10.5k) | 4.94 GB/s     (4.8k)

ZFS backend r/w IOPS: 100/100

fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 100.36 MB/s  (25.0k) | 398.71 MB/s   (6.2k)
Write      | 100.62 MB/s  (25.1k) | 400.81 MB/s   (6.2k)
Total      | 200.99 MB/s  (50.2k) | 799.52 MB/s  (12.4k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 703.95 MB/s   (1.3k) | 2.65 GB/s     (2.5k)
Write      | 741.35 MB/s   (1.4k) | 2.83 GB/s     (2.7k)
Total      | 1.44 GB/s     (2.8k) | 5.48 GB/s     (5.3k)

Not sure why for ZFS it didn’t work. Any input?

ruskofd · February 10, 2023, 11:53am

That’s normal, this is explained in the documentation of the ZFS driver : ZFS - zfs - LXD documentation (linuxcontainers.org)

Arni_Johannesson · February 10, 2023, 12:08pm

It’s not really explained in any concrete sense. I assume you are refering to this section:

Observing I/O quotas
I/O quotas are unlikely to affect ZFS filesystems very much. That’s because ZFS is a port of a Solaris module (using SPL) and not a native Linux file system using the Linux VFS API, which is where I/O limits are applied.

“unlikely to affect ZFS filesystems very much” - but there is some effect?

I think yes, because in our real-world production experience across many many systems, then we found that adding the IOPS limits to LXD profiles for ZFS backed containers - while not limiting IOPS in any real sense - did resolve frequent zfs deadlock issues on very busy systems.

Basically we observed time and again that very busy systems using a lot of disk I/O would cause zfs kernel modules to deadlock and the host machines had to be rebooted. After we added the IOPS limit to lxd config, these issues went away and have not resurfaced since.

Just leaving this experience here in case it helps somebody else out there … And well… It would be nice to know what effects if any this config has on zfs behind the scenes, as my clear feeling is that it does indeed do something - just not what it says on the tin

ruskofd · February 10, 2023, 12:15pm

That’s good to know

wociscz · October 2, 2023, 4:22pm

Just tested iops limiting again with newest Incus & ZFS packages & 6.5 Kernel (all provided by Zabbly repositories) and as OP wrote, no luck with ZFS iops limiting.
DIR & BTRFS backend works with iops limiting just fine/as expected.

So what is the “official” state - ZFS won’t be able to limit iops at all (even in the future)? And I ~know there are problems with different licenses and other technical aspects.