One FS Server - btrfs Performance

LXD
1

I am dreaming of a one filesystem server with zfs or btrfs that runs the host and the containers on that same filesystem.

Installing Ubuntu on ZFS is not trivial and zsys development has halted in favour of active directory stuff which is unfortunate, at least for me.
Btrfs can be set up in the installer easily when setting up my base image.
And now why I consider this again now: Running Docker inside of a LXD container - YouTube from @stgraber. Would be cool to be able to run docker containers with good storage performance for production use in Linux Containers. So everything is pointing to btrfs.

So now the big question comes into play as I run monolithic LEMP stack containers with for now custom ZFS volumes for MariaDB with recordsize=16k for that zpool subdir…

Can I get good database performance from btrfs?
What would be the best settings for this?

Of course I’ll have to do tests again but some pointers in the right direction would be useful.
For example I learned here that turning off COW on the subvolume of the mysql data would probably be more performant. But can I still snapshot the whole container doing that?

Thanks in advance!

2

I do not have much experience with databases, but we have been running docker within lxd containers on the btrfs filesystem, and the performance was not great. Some btrfs operations (btfs-cleaner) were often stuck in the kernel, possibly due to CoW. Also, there is a known issue with btrfs sub-volumes not getting deleted by docker, which gets exacerbated if lxd snapshots are used (the primary reason for using btrfs) or if disk quotas are enabled. Since then we switched the docker storage driver to overlay2 (on top of btrfs) and the performance was much better. Our primary use cases is hosting jupyter notebooks (jupyterhub) for teaching.

1 Like
3

It also worth looking to the comparison of different storage drivers. Particularly, note that btrfs does not support page caching. So if many containers run off the same image, memory consumption / cache performance could be an issue. I wonder if lxd has the same problem.

4

I’ve run LVM, ext4, xfs, btrfs and zpool on performance critical solutions (including databases) and I’ve found btrfs for real world applications to be similar in performance to ext4. Though I think ext4 wins out overall. I’ve found XFS to be lagging behind the others in most areas. I normally use LVM then assign logical volumes to btrfs/zpool/ext4 where they are needed for flexibility.

There is no perfect filesystem… They all have strengths and weaknesses…

Though most of my low latency solutions the idea of using an off the shelf SQL or NoSQL database is not going to work out either.

My advice with filesystems is create your solutions in a flexible way (cloud-init) etc. and then benchmark the actual solutions using different filesystems and configuration and then use the path of least resistance.

For instance for LXD instances I’d use zpool unless I needed to use Docker in which case I’d use BTRFS. Though I still use ext4 for simplicity sake if a few seconds more spin up time isn’t an issue. Lets face it all very nice have near instant zpool/btrfs CoW spin up time but if your then waiting for 30 seconds for cloud-init to finish there isn’t much point.

1 Like