The official Incus documentation will helpfully tell you what you should prefer:
you should never use VMs with Btrfs storage pools.
That’s because:
Btrfs doesn’t natively support storing block devices. Therefore, when using Btrfs for VMs, Incus creates a big file on disk to store the VM. This approach is not very efficient and might cause issues when creating snapshots.
BTRFS OS Disk Subvolume for /var/lib/incus - #8 by stgraber :
With VMs being backed by a single file changing constantly we instead have to mark that file in btrfs as
nocow, effectively turning off the most useful part of btrfs. That makes things like snapshots use a rather less optimized codepath than what you’d get on containers.For those running a lot of VMs on local storage, your best options are ZFS or LVM.
OTOH, ZFS offers both file and block storage (datasets and zvols): this allows ZFS to expose a storage slice taken from its pool as a block device, that you can then format as you see fit with ext4, Btrfs etc. Incus creates zvols for VMs, which makes it much more efficient.
IncusOS is based on ZFS and Incus takes advantage of a lot of its features.
ZFS is much more flexible than Btrfs, and even though its learning curve is high, the rewards are worth it IMO.
Btrfs is fine on simple systems (I use it on my laptop and on some external storage), but ZFS is the enterprise-class storage.
Check out ZFS Basecamp - Klara Systems and https://discourse.practicalzfs.com/ for some help mastering it.