Is there any substantial advantage (one way or the other) btwn the Incus Host OS using BTRFS or ZFS?

21

I can confirm from personal experience that Btrfs trips over itself as soon as you go over ~10 snapshots.

I don’t know what you’re doing but I know that I’ve had 30 to 40 snapshots at any one time. I have automatic snapshots for every boot, every apt install or apt remove, anytime I make a major configuration change to something etc

22

This thread hopefully will be a source for a lot of good info for others
due to all the comments that have occurred so thanks to all that contributed!

I ​decided on a hybrid ​solution to satisfy both Incus CNs and VMs storage in a COW filesystem:

My only real driver for looking into zfs ​in the 1st place was to ​plan for any future ​increased
use of Incus VMs.

​To date, I use far more Incus Containers than VMs but I ​initially thought I ​should see how to

install/config/manage zfs ​to do my work​ so I posted this question on the forum.

After reading​ the various responses and after many related search results & answers
to questions​ I’d had, I decided to keep​ using Ubuntu on BTRFS w a flat-subvolume structure
​utilizing a flat subvol structure (@, @home, @incus, @container, @opt, @var etc)​, w Incus
and ​continued use of Snapper ​& the​ gui btrfs-assistant to manage/utilize the BTRFS
snapshot/restore for system and subvolumes.

​So ​I adopted a hybrid storage approach.

​On a separate 2TB nvme I created a 1TB zvol and a 1 TB btrfs partition

I reinstalled Incus and during init named the storage pool “cn-default” on the btrfs partition of that 2nd NVME.

Then used Incus to create an Incus ZFS pool for Incus VMs​ using the 1TB zvol:

$ incus storage create vm-default zfs source=/dev/nvme1n1 zfs.pool_name=vm-default

image
image1641×314 96.4 KB

Additional decision input for me came from reading several of stgraber’s comments in this 2025 thread resulting from a question someone posted ​who seemingly had ​similar goals as ​myself:

BTRFS OS Disk Subvolume for /var/lib/incus

​In my use, this hybrid solution​ should provide the best of both worlds w support​ing:

  • minimal learning curve as I’ve been using btrfs for years now and since btrfs is
    in-kernel, I can continue use of common linux tools and the snapper & btrfs assistant
    application combination
  • continue using current grub-btrfs to allow boot from snapshots at system boot
    Yes I know there is a similar functionality for zfs but I don’t have to make that change
  • importantly, my use gets the benefit of Incus:
    CN snapshot/restore on btrfs
    VM snapshot/restore on ZFS (minimizing impact of COW w VMs)

​This approach also allowed me to avoid a couple further decisions regarding:

1 Like
23

I don’t know what you’re doing but I know that I’ve had 30 to 40 snapshots at any one time. I have automatic snapshots for every boot, every apt install or apt remove, anytime I make a major configuration change to something etc

Snapper compounds the issues around Btrfs snapshots by firing all its snapshot-taking and pruning at exactly the same time, without any delay.

So the issue wasn’t really the number of snapshots, but that all 20-30 of them were updated/destroyed at exactly the same time.

Keeping them around 10-15 has reduced my issues with it, which is in line with what’s recommended in this Btrfs mailing list thread from 2018.

24

@neitsab

Snapper compounds the issues around Btrfs snapshots by firing all its snapshot-taking and pruning at exactly the same time, without any delay.

So the issue wasn’t really the number of snapshots, but that all 20-30 of them were updated/destroyed at exactly the same time.

in case you weren’t aware… you can do both selective “snapshots” and selective “restores”

Snapper uses configs to determine which subvolume to target and how often to take snapshots. By creating isolated configurations, you can snapshot different parts of your disk selectively