Is there any substantial advantage (one way or the other) btwn the Incus Host OS using BTRFS or ZFS?

1

I’m agnostic on filesystems but on my Incus Host machines, I have always used BTRFS.
Long ago I used ZFS but that was before my working with LXD & Incus.

Lately, I’ve been searching if perhaps there is a significant reason when using Incus for using
either ZFS or BTRS that I’m not be aware of.

I’ve used LXD & now Incus for a long time and my Host Ubuntu filesystems have always been BTRFS.

For the Host both ZFS and BTRFS promote many of the same advanced file system features

  • snapshots (automatic or manual)
  • BTRFS sub-volumes or ZFS filesystem Datasets
  • Rollback
  • Compression
  • COW

But there are differences to consider also!

BTRFS features, which are better than ZFS:

  • BTRFS does support shrinking a partition but decreasing the size of a ZFS pool requires
    creating a new, smaller pool and migrating the data
  • can be expanded by 1 drive at a time
  • online convert RAID profile (including single and DUP)
  • part of Linux kernel - upstream, supported and no potential license issue as w ZFS
  • support for OverlayFS

ZFS has an edge over BTRFS regarding:

  • RAIDZ has no write-hole issue as w BTRS Raid 5/6
    Note:
    This fact “may” be an important advantage of ZFS for some but I don’t see many screaming for BTRFS Raid 5/6 to be stabilized
  • ZFS has advanced cache tooling (however, with the death of Intel Optane, SLOG has become less used).
  • ZFS has been around longer & more tested
  • advanced optimizations such as per dataset compression

Incus VMs and COW Filesystems:

Running Incus VMs on COW filesystems like BTRFS and ZFS, the performance impact seems to have been reasonably addressed by both filesystems.

ZFS performs a bit better w VMs due to large caching support but at expense of greatly inceased memory utilization

BTRFS has fast VM performance but to prevent COW caused fragmentation the NOCOW attribute is used on VM images, Incus when it detects BTRFS automatically insures the COW is disabled for a new VM.

So my question is…
For an Incus focused use-case perspective is there a significant reason to use 1 or the other?

Or… in today’s world does it really matter any more?

2

ZFS has a much lower tendency to eat your data: that’s probably the most compelling reason.

If BTRFS has never eaten your data so far, you may not see this as a reason - and you won’t, until the day it happens.

In any case, whatever filesystem you use, you need to keep completely separate backups of your data (and snapshots are not backups).

3

Btrfs has quite a lot of quota issues as well.

4

Thanks for the response… Actually I was looking to find out if there was some ZFS capability that Incus takes advantage that is unique.

I thought I’d ask the question since I’ve recently created a separate set of SSDs to use ZFS instead of btrfs. Now at boot I can choose which using either filesystem to test/work with.

I have no real practical use experience with ZFS.
So I wondered if there was something I wasn’t aware of & I wanted to learn.

btrfs itself, has never caused a data loss for me yet in about 10 yrs of use but there is always a 1st time, However, me.. I can be a dolt and have wreaked plenty of havoc on my own.

But you are right that it could & does happen!

To mitigate that I initially used an app called Timeshift for btrfs snapshots/restores but Timeshift is primarily designed for system btrfs snapshots/restores. It takes a little more configuration for Timeshift to snapshot/restore a home subvolume (“@home”). But Timeshift worked well for me creating full system snapshots or just a user-data snapshot.

But I wanted finer granularity so I switched to a combination of 2 apps, Snapper to perform the btrfs snapshots/restores, and btrfs-assistant for a GUI front-end to Snapper.

Today, my btrfs file system is organized with separate sub-volumes & btrfs-assistant is configured to automatically use a separate schedule to do snapshots for each more or less often:

FYI… the incus forum doesn’t like using “@” symbol as it thinks I’m referring to people and won’t permit more than 10 people in 1 post so below where your see (at) its a stand-in for the “@” symbol

[(at)]=“/”
[(at)home]=“/home”
[(at)log]=“/var/log”
[(at)cache]=“/var/cache”
[(at)libvirt]=“/var/lib/libvirt”
[(at)incus]=“/var/lib/incus”
[(at)flatpak]=“/var/lib/flatpak”
[(at)docker]=“/var/lib/docker”
[(at)wireguard]=“/etc/wireguard”
[(at)containers]=“/var/lib/containers”
[(at)machines]=“/var/lib/machines”
[(at)var_tmp]=“/var/tmp”
[(at)tmp]=“/tmp”
[(at)opt]=“/opt”

Snapshots from any of the above can be restored by selecting it in btrfs-assistant.

If I only needed to restore some deleted file or directory I can mount the appropriate sub-volume and then browse into it with Nautilus and copy/paste the file or directory I need back.

I also installed & am using grub-btrfs but zfs has the similar capability w Zsys I think.

grub-btrfs improves the grub bootloader by adding a btrfs snapshots sub-menu, allowing me to boot into snapshots.

grub-btrfs supports manual cli created snapshots as well as Snapper and Timeshift created snapshots and it:

  • Automatically lists snapshots existing on the btrfs root partition.
  • Automatically detect if /boot is in a separate partition.
  • Automatically detect kernel, initramfs and Intel/AMD microcode in /boot directory within snapshots.
  • Automatically create corresponding menu entries in grub.cfg
  • Automatically detect the type/tags/triggers and descriptions/comments of Snapper/Timeshift snapshots.
  • Automatically generate grub.cfg if you use the provided Systemd

However, you are absolutely correct snapshots are not backups!

For safety, I can use btrfs’s send/receive to transfer btrfs volumes anywhere a portable 4TB ssd and offsite so they can be restored from there later or even imported to a different machine.

Lastly, I may be old now but over the years I have never trusted Larry Ellison & Oracle.
They may have never legally enforced their ZFS Copyright.
But that doesn’t mean they wouldn’t if enough $$$ were involved.

Again… thanks!

5

It’s Markdown, so put three backticks (```) on a line of their own, before and after the block of text you want to quote, and it will be preformatted as code and without special interpretation.

ZFS has an extremely permissive licence, moreso than GPL, which means that if enforcement came to pass, it would have to be Linus Torvalds against anyone who links ZFS into the kernel - for GPL violation.

Canonical took the view quite a few years ago (sensibly IMO) that this was unlikely to be an issue, and so supply pre-built ZFS kernel modules. In my book it’s the main reason for running Ubuntu rather than Debian.

6

Also, it’s interesting to note that openzfs (which is what’s usually meant by ZFS) continues to receive significant support from the US Department of Energy. I’m certain significant legal review occurred before DOE started using what’s now openzfs, let alone contributing back to the project. IANAL, but I doubt Oracle would make a fuss at this point in time.

7

The main reason I always avoided ZFS (as I’m usually in Debian environments) was needing to ensure the kernel module is built. I’ve spent a lot of time recovering systems where it wasn’t yet properly in place :@

However I’m thinking about using it again , this time only for incus’ storage. Thats because of the quota issues mentioned above - in particular containers reporting the full filesystem size instead of their allocation of it.

Its unlikely to be a compelling reason for lots of people but I’ve certainly reached the point where the pros and cons have matched up!

8

Thanks for the info !

9

I think the main zfs capability is really just quota limits, which are enforced.

The biggest issue with quotas on btrfs is a bad design right from the start. I’m not sure what the current situation is with bugs, but even if they are all fixed, breaking through a btrfs quota is as simple as creating a subvolume without a quota in your container. At least that’s what I remember from an old Stephan Graber video (the ones from the lxd days).

If you have no plan for using quotas, then I think btrfs has more advantages than disadvantages. I find snapper amazing and handling incus snapshots is easier in btrfs than in zfs itself. So I use btrfs on all my personal projects, as I don’t have to deal with any issues of having zfs out of tree.

Only use zfs now due to IncusOS, but I would still prefer btrfs there if it was available.

10

The official Incus documentation will helpfully tell you what you should prefer:

you should never use VMs with Btrfs storage pools.

That’s because:

Btrfs doesn’t natively support storing block devices. Therefore, when using Btrfs for VMs, Incus creates a big file on disk to store the VM. This approach is not very efficient and might cause issues when creating snapshots.

BTRFS OS Disk Subvolume for /var/lib/incus - #8 by stgraber :

With VMs being backed by a single file changing constantly we instead have to mark that file in btrfs as nocow, effectively turning off the most useful part of btrfs. That makes things like snapshots use a rather less optimized codepath than what you’d get on containers.

For those running a lot of VMs on local storage, your best options are ZFS or LVM.

OTOH, ZFS offers both file and block storage (datasets and zvols): this allows ZFS to expose a storage slice taken from its pool as a block device, that you can then format as you see fit with ext4, Btrfs etc. Incus creates zvols for VMs, which makes it much more efficient.

IncusOS is based on ZFS and Incus takes advantage of a lot of its features.

ZFS is much more flexible than Btrfs, and even though its learning curve is high, the rewards are worth it IMO.

Btrfs is fine on simple systems (I use it on my laptop and on some external storage), but ZFS is the enterprise-class storage.

Check out ZFS Basecamp - Klara Systems and https://discourse.practicalzfs.com/ for some help mastering it.

3 Likes
11

US DOE involvement is almost guaranteed to be because of the folks over at the Lawrence Livermore National Laboratory which, if i haven’t misremembered my OpenZFS history, were the creators of the ZFS port to Linux and later on, with the other ZFS projects, the grand unification into OpenZFS

1 Like
12

@neitsab
Thanks so much. That’s the kind of answer & info I had hoped to see.
Your comments actually confirmed much of what my own research had indicated but primarily “for VMs Btrfs doesn’t natively support storing block devices” which I was aware of.

As I’d said, I’ve used btrfs for a long time but just created a server w ZFS to gain some experience with using it, the various ZFS terminologies/acronyms, and what ZFS tools are available & how they work.

One thing your post did say that I’d not really understood yet was your statement:
" ZFS offers both file and block storage (datasets and zvols)"

As I’ve just started using zfs, and like any technology, understanding all the “terminology” comes with time. Although I’d seen mention of both “datasets” and “zvols” I was wondering about the difference and application “use-case” for each.

And although I started with LXC years ago, then used LXD and now Incus, I’ve not tried Incus OS yet as its a fairly recent addition. But its good to know Incus OS only uses ZFS as that is a key decision point for me to know.

Thanks!

1 Like
13

Not the person you were replying to, but hope the following helps you out.

You can think of a dataset as a regular directory on your filesystem, it can be manipulating like any other directory with all the bells and whistles (assuming that you have set the correct property flags) while a zvol is the logical equivalent to .qcow2 or .raw; it acts as a virtual block device, like another disk on your computer.

Nothing really prevents you from using them in roles that the other performs, it’s mostly a matter of ergonomics and performance overhead. For example, you wouldn’t really want to format and mount a zvol to do what a dataset could do on the same machine for example, but there’s nothing explicitly wrong with doing so aside from performance overhead.

If I’m remembering correctly, zvols were historically used as backing stores for iSCSI targets but nowadays have also largely seen usage for blocks storage in virtual machines.

In terms of Incus, it’s used as both the root disk for virtual machines and presenting additoinal disk devices to said VM.

That said, it used to be the case that zvols were, depending on benchmarking and platform specifics, strictly worse in terms of performance than a .qcow2 or .raw disk on a dataset; there have been PRs which have landed in OpenZFS upstream in recent times to alleviate these issues and make them perform similarly to each other. As to whether or not this holds true is something I haven’t heard others say nor have I had the time to test myself to say if it holds in my homelab.

1 Like
14

A practical use case for a zvol is running docker inside an incus container.

You can use a custom volume for /var/lib/docker and make that a zvol with zfs.block_mode: "true". So docker would relate with it as a typical ext4 filesystem.

15

The only real advantage BTRFS has over ZFS IMO is its lack of llicensing issues for Linux.

BTRFS RAID is crappy, unreliable and doesn’t fail over gracefully like RAIDZ does. I wouldn’t even consider using BTRFS for RAID personally. Its fundamentally broken.

ZFS’ performance is head and shoulders above BTRFS is nearly every respect ESPECIALLY when it comes to snapshots. BTRFS performance starts to significantly degrade when you have more than about 10 snapshots on a subvolume whilst ZFS is quite happy with thousands of snapshots per dataset.

1 Like
16

@bmullan My pleasure, happy that my answer could be useful to you!

As I’d said, I’ve used btrfs for a long time but just created a server w ZFS to gain some experience with using it, the various ZFS terminologies/acronyms, and what ZFS tools are available & how they work.

I’m in the same boat as you: long time Btrfs user who got more and more intrigued after hearing all the praises sung about ZFS, and it being used a lot in the home server space.

It took me a while to figure out the terminology, and I can recommend the following resources on that front:

As I’ve just started using zfs, and like any technology, understanding all the “terminology” comes with time.

Yeah, ZFS is definitely its whole world unto itself… Take it slow and steady!

2 Likes
17

Note that this is not necessary since OpenZFS 2.2 which added support for overlayfs.

Docker works perfectly fine in an Incus container on ZFS in its default configuration: no ZFS delegation and no changing Docker storage driver needed. Docker just uses the recommended and default overlay driver which openZFS now supports.

Using a ZFS block volume incurs the performance penalty which @SirGiggles elaborated on in his reply, which may or may not be significant depending on the workload but cannot be null.

18

Yep, true on all fronts: I didn’t mention it initially but the one big issue ZFS has on Linux is its licensing, which prevents its mainlining in the kernel and makes vendors wary of shipping its module by default (IIRC Canonical and Proxmox are the only major companies who are currently doing so).

I have never dabbled with Btrfs RAID for lack of need, here’s what its creator Chris Murphy had to say about it in 2024:

Yeah, Btrfs built-in raid is it’s own thing. Btrfs raid 0, 1, 1c3, 1c4, raid10 are considered stable. Btrfs parity raid (raid56 should probably be pronounced “raid five and six”, not “fifty-six” :grinning_face:) is experimental, and while it’s not inherently unsafe, you need to be extra familiar with its idiosyncracies to make sure your workload and expectations can tolerate it.

Note that “considered stable” is different from “in widespread production use”… I believe Synology (which uses Btrfs for its NASes) use their own RAID technology and not the native Btrfs one, so yeah it doesn’t inspire great confidence. ZFS RAIDz is known to be rock-solid.

I can confirm from personal experience that Btrfs trips over itself as soon as you go over ~10 snapshots. I’ve learnt that the hard way when I started using Snapper on my workstation, and scheduled snapshot creation/pruning would freeze the entire thing when there were too many…

So yeah, if ZFS was included by default in more distros that’s what I would use, and for professional storage there’s no need to look back.

19

Alternatively, you can consider this to be an issue in the licensing of Linux. If you use FreeBSD (which has a much more permissive licence than Linux) then ZFS is fully integrated, with no licence issues.

But if you use FreeBSD, you don’t get incus :frowning:

20

This probably veers off topic, but Synology uses a combination of md, lvm2, and one of btrfs or ext4 going by their support page: How can I use a PC to recover data when my Synology NAS malfunctions? - Synology Knowledge Center

My understanding is also that they’ve custom modified one of the layers in their stack to report errors to another but I can’t currently substantiate.

That should give some indication as to how reliable they consider Btrfs’ native RAID56 profiles.

1 Like