I noted that ZFS has taken over the default position from btrfs as filesystem for LXD storage pools.
Just out of curiosity I would like to know what the motivation behind this change was. Can anyone educate me on the motivation behind this?
to put my inquiry into context …
With very little insight in the technical details I decided a while ago to go with btrfs over zfs.
Reasons for this where:
btrfs is linux native
zfs, as per my understanding, is more RAM hungry
licenses
just to further clarify … I am not looking for a btrfs vs zfs discussion here, even though I would not mind at all to learn more on this from anybody with better understanding of the matter
ZFS has a very special place in Ubuntu, and will (likely) be the filesystem of choice.
Future versions of Ubuntu Desktop will be able to be installed and boot on ZFS. In Ubuntu 20.04/20.10 you can try it out already as an experimental feature. I have tried it and used it a lot. The cute outcome is that you can just create a dataset for LXD and tell lxd init to use that dataset. No need for extra partitioning!
Obviously, apart from Ubuntu Desktop, you can do the same with Ubuntu Server, etc.
Ever since we added support for multiple storage pools in LXD 2.1 or so, zfs has been our default pick. What has changed is that in the past zfs would need you to install the zfsutils-linux package in order for LXD to be able to use it, without that package, we’d default to btrfs.
With the LXD snap, the ZFS tools are always present, so as long as your kernel supports ZFS, we’ll default to it now.
Systems that would have defaulted to btrfs were systems where either the zfs kernel module or the zfs tools couldn’t be found during lxd init. zfs is always listed as the first option when available and so is picked as the default by lxd init.
LXD as pre-installed on Ubuntu 18.04 does not have the zfs tools available despite the kernel supporting it, so those systems and other prior Ubuntu releases would default to btrfs unless the user had preinstalled the zfs tools.
All releases after that use the snap which comes with the tools built-in, so on any LXD snap install, if the kernel has ZFS support, LXD will default to it during init.
Without being an expert on file systems, I got to the same question: btrfs or ZFS? The documentation says that btrfs has two features that ZFS does not have: storage driver usable inside a container and restore from older snapshots (not latest). On the other hand, the documentation recommends using ZFS:
The two best options for use with LXD are ZFS and btrfs.
They have about similar functionalities but ZFS is more reliable if available on your particular platform.
I would expect a reference in the documentation supporting the statement regarding the better reliability of ZFS.
We’ve never had a production user lose data on ZFS, we have had many production users lose data on btrfs.
Btrfs can be made somewhat reliable when you stay away from compression and RAID features but even then, its completely broken quota support makes it a no-go for any environment where you need quotas to actually be enforced.
In my use case I stay away from compression and RAID. Also, I do not need support for quotas because I use the computer to run benchmark and containers to improve the repeatability of the experiments. Thus, because the aforementioned two additional features, I preferred to use btrfs.
I guess that the feature of being the storage driver usable inside a container is relevant to use nested containers. Otherwise, the nested containers would not take advantage of the storage driver optimizations. Am I right?
I have deployed LXD on low-end commodity hardware. At the moment I have 4 machines with Ubuntu20.04 server on the bare metal with LXD installed using all defaults. The four machines are successfully clustered.
Two machines have a 16GB RAM; two machines are at 8BG RAM. Each has at least one dedicated HD.
My goal is to setup a development environment and tool train in my garage.
I can dedicate another machine strictly for ZFS as NAS; what I do not know is: Is doing so a best practice; or is the best practice to have a dedicated disk and/or partition at each local node.
LXD would expect a local pool on each machine. The only remote storage option which LXD can use is Ceph, but that is similarly distributed among many machines.
I also use LXD containers for testing and development (not for selling containers to customers). The so called “broken” quota support, seems to be a feature in my case.
I also need to install Docker inside the containers. The docs state clearly that:
Docker will not run well with the default zfs file system
and:
Btrfs is one of the storage pools Docker supports natively
So, it is clear that ZFS is not better than Btrfs in all the cases. Maybe the docs should explain the advantages and disadvantages better, so that the users can make an informed choice.
@stgraber has done a series of videos covering each of the lxd storage drivers along with a discussion of their pros and cons you may find helpful.
Keep in mind also that because lxd supports custom volumes and these can be attached to instances at particular Mount points, it means you can use different storage pools within a single instance to best suit the particular requirements of a directory.
For example, you could have your docker directory on a btrfs, dir or lvm (with ext4 for overlayfs support and effective quotas) custom volume , and the rest of the container on zfs.
Docker will not run well with the default zfs file system