ZFS from lxc write speed slower than native

Hey there!

I’m playing with Ubuntu 18.04.3 on a Raspberry Pi 3B+.
My storage is a new 240GB Kingston SSD in a USB enclosure, entirely dedicated to act as lxc storage.
I do my tests currently not within a container but on the host.
As a quick performance test, I run dd if=/dev/urandom of=tmpfile bs=1MB count=1024.

I encounter a massive performance drop on storages created by lxc.

  • If I create a native zpool with zpool create ssd /dev/sdc1
    then I get a write performance between 28MB/s and 30MB/s.
  • If I create a zpool with lxc storage create ssd zfs source=/dev/sdc1
    then I get a write performance around 10MB/s.

I tried to compare both, zpool and zfs attributes between natively created and by lxc. There’s not much difference and even if I manually adjust those by zfs set or zpool set, I can’t find out why exactly the lxc handled zfs is only 30% of the performance of a native ZFS.

Is this to be expected and there’s a reasonable explanation to it?

Regards,
Stephan.

Can you test again but making sure to pass conv=fdatasync?

Running dd without having it force the data be written to disk would give you very varying results depending on the amount of memory available and when zfs decides to then flush writes to disk in the background.

Hi Stéphane.

Thank you for your response. You’re right, of course fdatasync has to be used when testing dd.

I did my tests again with conv=fdatasync, as a command of dd if=/dev/urandom of=tmpfile bs=1MB count=1024 conv=fdatasync in total.

  • If I create the pool via zpool create ssd /dev/sdc1
    then the dd command makes from 25MB/s to 28MB/s.
    That’s slightly slower than a couple of hours ago, but I expect the USB performance to be to a certain degree unstable.
  • If I create the pool via lxc storage create ssd zfs source=/dev/sdc1
    then the dd command goes down to 10MB/s.

So that’s basically the same result, conv=fdatasync doesn’t make a difference.

I digged deeper.

  • If I do something like this:
    zpool create ssd /dev/sdc1
    lxc storage create dir source=/ssd/directory
    then I get 28MB/s even inside a container.
  • If Ido something like this:
    zpool create ssd /dev/sdc1
    zfs create ssd/storage
    lxc storage create ssd zfs source=ssd/storage
    then I get 28MB/s in /ssd
    but only 10MB/s in /var/lib/lxd/storage-pools/ssd/

So I suspect its not an attribute of the zpool but either an attribute of the zfs or a mount option.

Regards,
Stephan.

You can run zfs get all DATASET to compare the options set in either case and see what may cause that impact in your case. I suspect that compression may be part of the issue as it’s usually quite beneficial, but on a USB drive where IOps are expensive, the added roundtrips due to the smaller writes could explain the difference.

Hi Stéphane.

That’s what I meant by comparing zpool and zfs attributes in my initial post.

Here’s what I do to compare:

zpool create ssd /dev/sdc1
zfs create ssd/storage-native
zfs create ssd/storage-lxc
lxc storage create ssd zfs source=ssd/storage-lxc
zfs get all ssd/storage-native | awk '{ print $2 ": " $3 "(" $4 ")" }' > native
zfs get all ssd/storage-lxc | awk '{ print $2 ": " $3 "(" $4 ")" }' > lxc
diff -ru native lxc

The differences are:

--- /root/native        2019-08-24 10:35:54.611812227 +0200
+++ /root/lxc   2019-08-24 10:36:12.591708955 +0200
@@ -1,29 +1,29 @@
 PROPERTY: VALUE(SOURCE)
 type: filesystem(-)
 creation: Sat(Aug)
-used: 24K(-)
+used: 144K(-)
 available: 215G(-)
 referenced: 24K(-)
 compressratio: 1.00x(-)
-mounted: yes(-)
+mounted: no(-)
 quota: none(default)
 reservation: none(default)
 recordsize: 128K(default)
-mountpoint: /ssd/storage-native(default)
+mountpoint: none(local)
 sharenfs: off(default)
 checksum: on(default)
 compression: off(default)
 atime: on(default)
-devices: on(default)
-exec: on(default)
-setuid: on(default)
+devices: on(local)
+exec: on(local)
+setuid: on(local)
 readonly: off(default)
 zoned: off(default)
 snapdir: hidden(default)
 aclinherit: restricted(default)
-createtxg: 12(-)
+createtxg: 14(-)
 canmount: on(default)
-xattr: on(default)
+xattr: sa(local)
 copies: 1(default)
 version: 5(-)
 utf8only: off(-)
@@ -34,12 +34,12 @@
 sharesmb: off(default)
 refquota: none(default)
 refreservation: none(default)
-guid: 11452761463892599915(-)
+guid: 15648986131708524059(-)
 primarycache: all(default)
 secondarycache: all(default)
 usedbysnapshots: 0B(-)
 usedbydataset: 24K(-)
-usedbychildren: 0B(-)
+usedbychildren: 120K(-)
 usedbyrefreservation: 0B(-)
 logbias: latency(default)
 dedup: off(default)
@@ -48,7 +48,7 @@
 dnodesize: legacy(default)
 refcompressratio: 1.00x(-)
 written: 24K(-)
-logicalused: 12K(-)
+logicalused: 72K(-)
 logicalreferenced: 12K(-)
 volmode: default(default)
 filesystem_limit: none(default)
@@ -56,7 +56,7 @@
 filesystem_count: none(default)
 snapshot_count: none(default)
 snapdev: hidden(default)
-acltype: off(default)
+acltype: posixacl(local)
 context: none(default)
 fscontext: none(default)
 defcontext: none(default)

Even i I turn the acl off by zfs set acltype=off ssd/storage-lxc and set zfs set xattr=on ssd/storage-lxc as to have the diff only show some readonly used bytes as differences, the performance still differs by a factor of 2 to 2.5. That’s kind of what I expected, beause ACL checking on a single 1GB file should not take place for every byte but only once for the file :).

Compression and dedup are off by default.

Thank you for your help!

Regards,
Stephan.

Hey there!

I guess this thread can be closed, partially due to hardware issues, partly due to not checking the actual mount point.

I have one SSD which currently hosts a couple of running containers. I can’t get this one to perform better than 10MB/s, but testing capabilities are limited due to it homing those containers.

I have another SSD. That’s the one I’ve been testing here in this thread. So I constantly created and dropped zpools, zfs as well as lxc storages. That one didn’t get above 10MB/s as well in different LXC scnearios but close to 30MB/s without LXC.

I thought it would be the same thing as the first SSD since the 10MB/s limit kicked in as soon as lxc got involved.
It wasn’t.
The actual cause was: I didn’t realize the zfs wasn’t mounted to /var/lib/lxd/storage-pools/$poolname but to /var/lib/lxd/storage-pools/$poolname/containers/$containername. So whenever I tested a dd last week, I actually filled the sd-card of my raspberry instead of the zfs ssd.

So:

  • My first SSD might be faulty and can’t get above 10MB/s at all.
  • The second SSD works just fine and ZFS performs as expected with and without lxc.
  • The 10MB/s of my test results weren’t SSD performance but sd-card performance. Clumsy me.

Thank you Stéphane for wasting your time on me.

Regards,
Stephan.