Hi
When i create a container using “lxc launch ubuntu:16.04 ct” R/W speed inside of the ct is very good.
But when i use below command to limit storage size, R/W speed slows down.
lxc config device add ct root disk path=/ pool=default size=2GB
Before add root device:
83886080 bytes (84 MB, 80 MiB) copied, 0.125395 s, 669 MB/s
after add root device:
83886080 bytes (84 MB, 80 MiB) copied, 5.58901 s, 15.0 MB/s
What storage backend do you use?
storage backend is zfs
In a loop device or on its own partition?
I created a partition for zfs
I have 3x2TB hdd and using raid5
What did you use to test the speed?
The 670MB/s is unrealistic for a 3x2TB RAID5 array so it’s almost certainly cached (non-flushed) writes and so the number would vary widely depending on the amount of RAM that’s available for caching and the constraints on the data (such as that quota).
That 15MB/s is much slower than I’d expect on such an array though, you should still be able to get 100MB/s from your setup, assuming nothing else is reading or writing data at the same time.
dd if=/dev/zero of=blah.img bs=4M count=50 conv=fsync
This will write 200MB of data to your disk, making sure that everything is actually written onto the drive rather than just kept in memory. One caveat though is that this is only reliable if compression is disabled, if it’s enabled, then you should use a source other than /dev/zero as /dev/zero compresses as well as possible and so will get you a much higher transfer speed than your drive is actually capable of.
Thank you stgraber for your response.
In host:
dd if=/dev/zero of=blah.img bs=4M count=50 conv=fsync:
209715200 bytes (210 MB, 200 MiB) copied, 3.47913 s, 60.3 MB/s
In Container:
209715200 bytes (210 MB, 200 MiB) copied, 45.1907 s, 4.6 MB/s
Compression is disabled
And your in host test is also using a dataset on that same zpool?
No my host has another partition.
r/w speed inside of the container is same as the host until container has not any storage limitation by below command:
lxc config device add ct root disk path=/ pool=default size=2GB
I created another zfs pool with ashift=12 and there is no difference
So you say the speed difference only appears after setting a size limit. Do you see that with a much larger size limit too?
All the LXD does when you set a size limit is apply it as a quota in ZFS so I’m wondering why ZFS is slowing down so much.
No @stgraber.
In larger size limits, speed is good.
I set size limit to 120GB and speed is the same as the host, Then i set it to 10GB and speed is the same as the host!
lxc launch is very slow about 30~50 seconds takes time to start.
This looks like the relevant report on ZFS,
Specifically, they say:
Poor write performance very close to the quota is a design choice made in
OpenZFS 7793 - ztest fails assertion in dmu_tx_willuse_space (3ec3bc2).
Hi
I think the problem is solved.