ZFS Pool Performance vs LXC

LXC
,
1

Hey All,

I’m cross posting two threads here in case someone might have any suggestions which may help me with a particularly peculiar problem.

Original Proxmox Thread: ZFS Pool Performance vs LXC | Proxmox Support Forum

Reddit Thread: https://www.reddit.com/r/Proxmox/comments/spne7t/zfs_pool_performance_vs_lxc/

After further testing and discussion on the above Reddit thread it appears as though the problem is not Proxmox related but something to do with LXC and ZFS.

A suggestion was made to compare the ZFS datasets. It seems the only difference between the tank and or my original LXC container (subvol-100) apart from the ones you’d expect to be different such as compressionratio objsetid guid etc are the below:

Value /tank/ /tank/subvol-100 (original LXC) /tank/subvol-138 (new LXC)
xattr on on sa
acltype off off posix

An overview of what’s happening, upon contemplating upgrading to 10GB I wanted to perform some testing on my pool to determine it’s performance. All tests were performed using fio using the following command fio --name=test --size=5g --rw=write --ioengine=posixaio --direct=1 --bs=1m

I learnt that when running in the root level of my zfs pool I was able to get around ~529MiB/s. When running the same test on my first lxc container the result was more or less the same. However only this particular lxc and another can achieve this, all my others show results in the ~215-248MiB/s range.

I fired up a secondary Proxmox server to try and determine if this was isolated to my primary server or not and on a fresh installation the same problem can be observed. The root pool always performs better than an lxc container.

Testing Performed:

  1. fio from my primary Proxmox host on the root zfs pool /tank. Results with a write of around ~529MiB/s
  2. fio from my primary Proxmox host on an lxc subvol-* folder inside /tank. On two lxc containers I receive similar results to the host, on all others only around ~215-248MiB/s write.
  3. fio from my newly created secondary Proxmox host on the root zfs pool /tank. Results with a write of around 440MiB/s (different pool vdev layout).
  4. fio from my newly created secondary Proxmox host on an lxc subvol-* folder inside /tank. On all created lxc containers results report only around 163MiB/s write.
  5. On my secondary Proxmox host I created a VM and with sync=disabled was able to achieve similar results to the root pool, 402MiB/s vs 438MiB/s on the host. With sync=standard results were similar to the performance numbers seen from the lxc container 149MiB/s.
  6. Results don’t change greatly if fio is performed from inside the host on the /tank zfs pool or directly on the lxc container.
  7. Secondary host was booted with mitigations=off as a troubleshooting step suggested from the Proxmox forums, no difference in performance.
  8. Creating an lxc unprivileged vs privileged makes no difference to fio tests.
  9. Creating a directory located on the /tank zfs pool and spinning up an lxc with a rootfs located on it with sync=disabled results in performance better than an lxc container but not close to the root zfs pool with a write of around ~324MiB/s.
  10. Creating a directory on the /tank zfs pool and mounting it inside a privileged lxc container results similar results to the root pool.

What I’m trying to understand is what is the difference between the two lxc containers I have on my primary Proxmox host which perform vastly different to any other lxc created on that same host or a brand new Proxmox host.

All testing and result outputs can be found in the link at top of this post.

I would appreciate any feedback or suggestions anyone can provide to help me narrow down what could be causing this problem.

Thanks in advance.