ZFS vs EXT4 Performance on High Load with LXD-Benchmark

LXD
, ,
41

You are unable to perform real-world testing and are limited to using synthetic benchmark tools ?

42

I canā€™t yet, I have to use these tools

But all the utilities Iā€™ve used before have shown bad results on ZFS

43

What result were you anticipating with this latest synthetic test and why ?

44

I expected faster ZFS speed than EXT4

As I wrote above because Canonical recommend using ZFS for LXD

PostgreSQL also recommend ZFS

45

Iā€™m asking what is the result (data) you anticipated with your most recent test and how is it different than what you anticipated (data) such as :
latency average = 25.720 ms
latency stddev = 13.184 ms
initial connection time = 89.406 ms

and you anticipated:

latency average = 15 ms
latency stddev = 8 ms
initial connection time = 50 ms

46

If we return to the topic of creating LXD containers, then I expected faster container creation on ZFS than on EXT4

300 containers per 13.215s on EXT4 VS 300 containers per 7.420s on ZFS

but not

300 containers per 13.215s on EXT4 VS 300 containers per 24.420s on ZFS

Or at least equal value but not like a double loss

Do you have any ideas about this?
Have you experienced this?

47

You are running synthetic benchmark tools and seem to have no expectation. Therefore, there seems no issue you are troublshooting and simply do not like the results are different than the video demo, correct?

48

Absolutely right

When using the same server, the speed of creating containers in ZFS is worse than on DIR

And I donā€™t know if this is the norm or not, because the video and other sources say otherwise

Regards.

49

Do you have any idea how to solve the ZFS speed issue when creating LXD containers?

Regards.

50

I donā€™t see a production issue, and synthetic testing without specific data-point expectations can quickly become a distracting rabbit holeā€¦

51

Ok i understand you
Thank you very much for your attention

Iā€™ll try to analyze it myself

Regards.

52

Iā€™ll let you know as soon as there are achievements.

If anyone has any suggestions I will be glad to hear!
Regards.

53

Hello everyone again!
I started another test
This time ZFS was 3x faster!

Filesystems behave differently in different workloads

300 containers per 17.562s on EXT4 VS 300 containers per 5.225s on ZFS

lxd-benchmark init --count 50 --parallel 10 images:debian/11
Test environment:
  Server backend: lxd
  Server version: 5.0.2
  Kernel: Linux
  Kernel architecture: x86_64
  Kernel version: 5.15.79-0-lts
  Storage backend: zfs
  Storage version: 2.1.6-1
  Container backend: lxc
  Container version: 5.0.2

Test variables:
  Container count: 50
  Container mode: unprivileged
  Startup mode: normal startup
  Image: images:debian/11
  Batches: 5
  Batch size: 10
  Remainder: 0

[Jun 23 12:42:27.321] Importing image into local store: 8294cda0215fdf3b78bab4f4149584592ad10e7a09bf9d5d0da4d10413d5aafc
[Jun 23 12:42:49.163] Found image in local store: 8294cda0215fdf3b78bab4f4149584592ad10e7a09bf9d5d0da4d10413d5aafc
[Jun 23 12:42:49.164] Batch processing start
[Jun 23 12:42:51.559] Processed 10 containers in 2.396s (4.174/s)
[Jun 23 12:42:52.358] Processed 20 containers in 3.194s (6.262/s)
[Jun 23 12:42:53.693] Processed 40 containers in 4.530s (8.831/s)
[Jun 23 12:42:54.389] Batch processing completed in 5.225s

lxd-benchmark init --count 50 --parallel 10 images:debian/11
Test environment:
  Server backend: lxd
  Server version: 5.0.2
  Kernel: Linux
  Kernel architecture: x86_64
  Kernel version: 5.15.79-0-lts
  Storage backend: dir
  Storage version: 1
  Container backend: lxc
  Container version: 5.0.2

Test variables:
  Container count: 50
  Container mode: unprivileged
  Startup mode: normal startup
  Image: images:debian/11
  Batches: 5
  Batch size: 10
  Remainder: 0

[Jun 23 12:47:54.530] Importing image into local store: 8294cda0215fdf3b78bab4f4149584592ad10e7a09bf9d5d0da4d10413d5aafc
[Jun 23 12:48:16.543] Found image in local store: 8294cda0215fdf3b78bab4f4149584592ad10e7a09bf9d5d0da4d10413d5aafc
[Jun 23 12:48:16.543] Batch processing start
[Jun 23 12:48:20.179] Processed 10 containers in 3.636s (2.750/s)
[Jun 23 12:48:23.620] Processed 20 containers in 7.077s (2.826/s)
[Jun 23 12:48:30.624] Processed 40 containers in 14.081s (2.841/s)
[Jun 23 12:48:34.105] Batch processing completed in 17.562s
54

ZFS should be slower than EXT4?

ZFS has built in integrity checks so it is giving more reliability than ext4, but the cost is reduced performance. They are not comparing the same thing.

1 Like
55

@tomp Just because of the capabilities of ZFS, I use it

Itā€™s an awesome file system especially when integrated with LXD

And with some changes it can run faster than EXT4

1 Like
56

Hi, itā€™s interesting what did you do to make it faster?
I see it is:
lxd-benchmark init --count 300 images:debian/11
vs
lxd-benchmark init --count 50 --parallel 10 images:debian/11
Is it the only change? Or some ZFS tuning?

57

Hi!

Actually I changed the workload type and also used some zfs settings

ZFS performed better with lxd-benchmark init --count 300 images:debian/11 than with lxd-benchmark init --count 50 --parallel 10 images:alpine/edge

Here are the settings I used for the test

bench:~# zpool create -f -o ashift=12 \
> -O acltype=posixacl -O canmount=off -O compression=lz4 \
> -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \
> -O recordsize=8K -O atime=off -O logbias=throughput \
> data mirror /dev/sda /dev/sdb mirror /dev/sdc /dev/sdd mirror /dev/sde /dev/
sdf mirror /dev/sdg /dev/sdh mirror /dev/sdi /dev/sdj mirror /dev/sdk /dev/sdl

Regards.

58

Thanks for sharing @Ibragim_Ganizade!

1 Like