I recently imported an old VM image from LXD into incus (used lxc export VM).
The VM disk image size was 256GB, but due to compression (I use compress=zstd mount option for btrfs) the actual used size on the host was ~ 90GB. The VM also had a “clean install” snapshot which I took when I did the initial setup and the disk image had about 8GB of used host space.
When I used incus import VM.tar.gz, I noticed something very strange: The snapshot + VM was taking the full 512GB of disk space as if ignoring the compression option I had set globally on btrfs. I confirmed this by copying the VM disk image to another BTRFS subvolume, which resulted in it using only the original 90GB.
I tracked the issue down to the “nodatacow” attribute on the VM subvolume, lsattr -d was showing C and found that it was added in December 2021, which was after I had created the VM (which is probably why my VM/snapshot never had this attribute on LXD): lxd/storage/drivers/driver/btrfs/volumes: Enable nodatacow on subvolu… · lxc/incus@2fee704 · GitHub . I would have reported this as a bug, but since it seems intentional I’d rather discuss it here.
I don’t know about other BTRFS users, but for me the “nodatacow” option does not make any noticeable difference in performance since I use NVME ssds. I think in the age of spinning disks this might have made a lot of difference, but nowadays random I/O is very fast and I’d rather have the advantages of compression.
To show much difference this makes in my case, here’s the output of df before disabling nodatacow on my imported VM
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 1.4T 264G 1.2T 19% /var/lib/incus
Notice I removed the snapshot, so the VM was using 256GB vs 512GB when I imported it. Here’s the output after disabling “nodatacow”, moving the disk into another subvolume and then moving it back (to force re-compression):
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 1.4T 62G 1.4T 5% /var/lib/incus
200GB less, a 75% reduction!
This can make a lot of difference to sysadmins which use LXD/Incus to implement their clouds, allowing them to over commit VM storage space to users.
Can we consider removing “nodatacow” or at least add a configuration option for this? It is possible to fix this manually by going into each VM subvolume and invoking chattr -C . (as I already done), but it would be much more convenient if this was an user preference and done automatically by Incus.