ZFS as default for storage pools - motivation

LXD
#1

I noted that ZFS has taken over the default position from btrfs as filesystem for LXD storage pools.

Just out of curiosity I would like to know what the motivation behind this change was. Can anyone educate me on the motivation behind this?

to put my inquiry into context …

With very little insight in the technical details I decided a while ago to go with btrfs over zfs.
Reasons for this where:

  • btrfs is linux native
  • zfs, as per my understanding, is more RAM hungry
  • licenses

just to further clarify … I am not looking for a btrfs vs zfs discussion here, even though I would not mind at all to learn more on this from anybody with better understanding of the matter

#2

ZFS has a very special place in Ubuntu, and will (likely) be the filesystem of choice.
Future versions of Ubuntu Desktop will be able to be installed and boot on ZFS. In Ubuntu 20.04/20.10 you can try it out already as an experimental feature. I have tried it and used it a lot. The cute outcome is that you can just create a dataset for LXD and tell lxd init to use that dataset. No need for extra partitioning!

Obviously, apart from Ubuntu Desktop, you can do the same with Ubuntu Server, etc.

It is important to learn more about ZFS, and here is a list of articles https://discourse.ubuntu.com/t/zfs-focus-on-ubuntu-20-04-lts-blog-posts/16355

Regarding licensing, see https://ubuntu.com/blog/zfs-licensing-and-linux

I wrote earlier that ZFS is a filesystem. That’s not precise enough. ZFS is a filesystem and volume manager.

#3

Ever since we added support for multiple storage pools in LXD 2.1 or so, zfs has been our default pick. What has changed is that in the past zfs would need you to install the zfsutils-linux package in order for LXD to be able to use it, without that package, we’d default to btrfs.

With the LXD snap, the ZFS tools are always present, so as long as your kernel supports ZFS, we’ll default to it now.

#4

that is new to me. I pretty much remember btrfs being the default until recently (or maybe it was only btrfs then)

#5

Systems that would have defaulted to btrfs were systems where either the zfs kernel module or the zfs tools couldn’t be found during lxd init. zfs is always listed as the first option when available and so is picked as the default by lxd init.

LXD as pre-installed on Ubuntu 18.04 does not have the zfs tools available despite the kernel supporting it, so those systems and other prior Ubuntu releases would default to btrfs unless the user had preinstalled the zfs tools.

All releases after that use the snap which comes with the tools built-in, so on any LXD snap install, if the kernel has ZFS support, LXD will default to it during init.

#6

Without being an expert on file systems, I got to the same question: btrfs or ZFS? The documentation says that btrfs has two features that ZFS does not have: storage driver usable inside a container and restore from older snapshots (not latest). On the other hand, the documentation recommends using ZFS:

The two best options for use with LXD are ZFS and btrfs.
They have about similar functionalities but ZFS is more reliable if available on your particular platform.

I would expect a reference in the documentation supporting the statement regarding the better reliability of ZFS.

#7

We’ve never had a production user lose data on ZFS, we have had many production users lose data on btrfs.

Btrfs can be made somewhat reliable when you stay away from compression and RAID features but even then, its completely broken quota support makes it a no-go for any environment where you need quotas to actually be enforced.

#8

In my use case I stay away from compression and RAID. Also, I do not need support for quotas because I use the computer to run benchmark and containers to improve the repeatability of the experiments. Thus, because the aforementioned two additional features, I preferred to use btrfs.

I guess that the feature of being the storage driver usable inside a container is relevant to use nested containers. Otherwise, the nested containers would not take advantage of the storage driver optimizations. Am I right?

#9

Hi all… I am still a relative newbie on LXD.

I have deployed LXD on low-end commodity hardware. At the moment I have 4 machines with Ubuntu20.04 server on the bare metal with LXD installed using all defaults. The four machines are successfully clustered.

Two machines have a 16GB RAM; two machines are at 8BG RAM. Each has at least one dedicated HD.

My goal is to setup a development environment and tool train in my garage.

I can dedicate another machine strictly for ZFS as NAS; what I do not know is: Is doing so a best practice; or is the best practice to have a dedicated disk and/or partition at each local node.

Thx in advance for your guidance.

Kind Regards,

M. Fene’

#10

LXD would expect a local pool on each machine. The only remote storage option which LXD can use is Ceph, but that is similarly distributed among many machines.