What filesystem to use for lxd?

LXD
#1

I’m trying to host websites inside lxd. The thing common to all websites is static content, php and database. The ideal scenario wud be a different container for every website. Some containers might just include file backups, some I’ll mount as drives on my laptop.

I’ve wanted to use lxd’s snapshot feature but I don’t want it at the cost of severely reduced io. Ideally if the server went down or I had to move to a bigger server, cud I restore a snapshot quickly on a different host server?

I’m worried that any wrong decision now will hurt me later. What is the ideal filesystem for a situation like this?

#2

Btrfs!
Why not ZFS?

  1. biggest issue is that it’s not mainlined and will never be (as far as we can tell)
  2. secondly it has a lot of features and abstraction layers for larger installations that will get in your way.
  3. Does not work with docker which I’m guessing you will be using with container nesting

NOTE
you should consider disabling Copy-on-Write for the directory that the database resides in for better I/O.
Don’t forget to enable scrubbing service for Btrfs

1 Like
#3

When I’ve run yabs.sh on btrfs I’ve observed a 15-20% degradation in read and write compared to the host. I don’t know why that is. I chose lvm and the io rate at times was even better than host. But I don’t know if in the long run any of the filesystems will suit my needs and if the degradation is actually relevant in the first place.

#4

Btrfs is being actively developed, you might wanna take a look at recent changes merged in linux 6.1 and 6.2

If future proofing is a priority then Btrfs is pretty much the way forward, Btrfs has become the default file system for most major distros and companies like synology have already started shipping many of their NAS products with btrfs out of the box.

#5

The default option for LXD is ZFS at the moment. I’ve been using it for a few years now without any problem. The performance is excellent (especially if you have a lot of memory).

You can also enable ZFS in regular Ubuntu desktop installations. ZFS vs BTRFS depends on your workload. I have docker in LXD VMs with ZFS storage. Btrfs is the best option for native docker support in LXD containers. ZFS is also the default in Proxmox. It is part of the kernel. I’ve never run into any limitations with ZFS. I usually adjust ashift to 12 for 4k blocks and sometimes the ARC memory cache limit.

Check the documentation for more information:

#6

I agree,

I used btrfs before. now i use zfs and the performance increase on zfs is great!

#7

I just found this interesting article which compares btrfs and zfs. RedHat does not seem to like either and is rooting for Stratis, a user-space daemon that coordinates various other disk management technologies (seems similar to what LXD is doing for containers).

The choice of filesystem impacts your procedures about backups, snapshots, etc. I am very happy with ZFS, and I use it more and more. Important ZFS features are mirroring (RAID), snapshots, incremental send/receive, compression, and encryption.

Make sure that your choice of database works well with the filesystem. I’ve adopted some performance tuning recommendations for the recordsize of zfs filesystems used by mariadb. Admittedly this increases the complexity of setting up the database as I need to attach two additional filesystems to the mariadb container (data, log), and configure mariadb to use them. If you do not do such special tuning, the performance may still be acceptable. Run a benchmark to compare running the database in a container or outside of LXD, with different filesystems. The “benchmark” that I use is timing the loading of a large .sql file, previously created with mysqldump.

I generally use a single debian-based mariadb container and have the website containers use that. I made an exception for a larger database, which I run on the same container as the website. My standard two-container configuration was causing this particular website/database to hang on certain database operations, for reasons that I have not fully understood. Because of this, I may remove the special mariadb tuning that I mentioned. Beware of too much optimization.