I’m cross posting two threads here in case someone might have any suggestions which may help me with a particularly peculiar problem.
Original Proxmox Thread: ZFS Pool Performance vs LXC | Proxmox Support Forum
After further testing and discussion on the above Reddit thread it appears as though the problem is not Proxmox related but something to do with LXC and ZFS.
A suggestion was made to compare the ZFS datasets. It seems the only difference between the tank and or my original LXC container (subvol-100) apart from the ones you’d expect to be different such as
guid etc are the below:
|Value||/tank/||/tank/subvol-100 (original LXC)||/tank/subvol-138 (new LXC)|
An overview of what’s happening, upon contemplating upgrading to 10GB I wanted to perform some testing on my pool to determine it’s performance. All tests were performed using
fio using the following command
fio --name=test --size=5g --rw=write --ioengine=posixaio --direct=1 --bs=1m
I learnt that when running in the root level of my zfs pool I was able to get around ~529MiB/s. When running the same test on my first lxc container the result was more or less the same. However only this particular lxc and another can achieve this, all my others show results in the ~215-248MiB/s range.
I fired up a secondary Proxmox server to try and determine if this was isolated to my primary server or not and on a fresh installation the same problem can be observed. The root pool always performs better than an lxc container.
fiofrom my primary Proxmox host on the root zfs pool
/tank. Results with a write of around ~529MiB/s
fiofrom my primary Proxmox host on an lxc
/tank. On two lxc containers I receive similar results to the host, on all others only around ~215-248MiB/s write.
fiofrom my newly created secondary Proxmox host on the root zfs pool
/tank. Results with a write of around 440MiB/s (different pool vdev layout).
fiofrom my newly created secondary Proxmox host on an lxc
/tank. On all created lxc containers results report only around 163MiB/s write.
- On my secondary Proxmox host I created a VM and with
sync=disabledwas able to achieve similar results to the root pool, 402MiB/s vs 438MiB/s on the host. With
sync=standardresults were similar to the performance numbers seen from the lxc container 149MiB/s.
- Results don’t change greatly if
fiois performed from inside the host on the
/tankzfs pool or directly on the lxc container.
- Secondary host was booted with
mitigations=offas a troubleshooting step suggested from the Proxmox forums, no difference in performance.
- Creating an lxc unprivileged vs privileged makes no difference to
- Creating a directory located on the
/tankzfs pool and spinning up an lxc with a
rootfslocated on it with
sync=disabledresults in performance better than an lxc container but not close to the root zfs pool with a write of around ~324MiB/s.
- Creating a directory on the
/tankzfs pool and mounting it inside a privileged lxc container results similar results to the root pool.
What I’m trying to understand is what is the difference between the two lxc containers I have on my primary Proxmox host which perform vastly different to any other lxc created on that same host or a brand new Proxmox host.
All testing and result outputs can be found in the link at top of this post.
I would appreciate any feedback or suggestions anyone can provide to help me narrow down what could be causing this problem.
Thanks in advance.