f-bn
February 29, 2024, 12:17pm
2
I remember at the time of LXD 4.20, there was a lot of discussions around problem with BTRFS quota and VM. I don’t know if it’s better know, but in the past, this wasn’t great
Yeah its not great the way BTRFS quotas work.
We added some documentation around the use of VMs on BTRFS quotas:
“Our recommendation is to not use VMs with btrfs storage pools…”
opened 08:48AM - 18 Aug 21 UTC
closed 05:55PM - 02 Dec 21 UTC
Bug
Even though LXD sets the BTRFS quota 100MiB (by default) larger than the VM disk… file max size, if the VM uses all of its disk space the underlying BTRFS filesystem sees that the referenced disk quota has been reached and prevents LXD from starting the VM because it cannot write the backup file. Even though there should be some space free.
It seems like BTRFS quota isn't working the way we think it does.
See https://discuss.linuxcontainers.org/t/btrfs-issues-storage-pools-btrfs-empty-and-btrfs-quota-100-while-inside-the-vm-only-48-utilized/11897
Steps to reproduce:
```
lxc storage create btrfs btrfs
lxc init images:ubuntu/focal/cloud v1 --vm -s btrfs
# This should do 2 things; increase disk image file size to 11GB, and enable BTRFS disk quotas to a size of 11GB+100MiB (which is the default `volume.state` size) to account for the entirety of the disk size, and allow 100MiB for volume state usage.
lxc config device set v1 root size=11GB # Accounting for VM image file size sizeBytes=11104862208 blockSize=11000004608
```
Check BTRFS quota set (expect it to be blockSize 11000004608 + 100MiB (104857600 bytes) = 11104862208 bytes):
```
sudo btrfs subvolume show --raw /var/lib/lxd/storage-pools/btrfs/virtual-machines/v1
virtual-machines/v1
Name: v1
UUID: 5816cce0-0dba-3348-bd29-14b80dfdbc85
Parent UUID: 06d1312f-5158-f44e-89a0-999e132ee7bc
Received UUID: -
Creation time: 2021-12-01 12:27:57 +0000
Subvolume ID: 272
Generation: 2516
Gen at creation: 2515
Parent ID: 5
Top level ID: 5
Flags: -
Snapshot(s):
Quota group: 0/272
Limit referenced: 11104862208 # Matches requested sizeBytes of 11104862208
Limit exclusive: -
Usage referenced: 2361466880
Usage exclusive: 57344
```
Check size of root disk file (expect it to be 11000004608 bytes):
```
sudo du -b /var/lib/lxd/storage-pools/btrfs/virtual-machines/v1/root.img
11000004608 /var/lib/lxd/storage-pools/btrfs/virtual-machines/v1/root.img
```
Start the VM:
```
lxc start v1
lxc shell v1
```
Now inside v1 run until the disk fills up (should fill up the disk image but not reach BTRFS quota as it has another 100MiB allowed):
```
cat /dev/urandom > /root/foo.bin
cat: write error: Read-only file system
```
You can now see that the BTRFS referenced quota has been reached, which it shouldn't have been.
```
sudo btrfs subvolume show --raw /var/lib/lxd/storage-pools/btrfs/virtual-machines/v1
virtual-machines/v1
Name: v1
UUID: 5816cce0-0dba-3348-bd29-14b80dfdbc85
Parent UUID: 06d1312f-5158-f44e-89a0-999e132ee7bc
Received UUID: -
Creation time: 2021-12-01 12:27:57 +0000
Subvolume ID: 272
Generation: 3246
Gen at creation: 2515
Parent ID: 5
Top level ID: 5
Flags: -
Snapshot(s):
Quota group: 0/272
Limit referenced: 11104862208
Limit exclusive: -
Usage referenced: 11104841728 # Limit reached, so how to explain if disk size is still 11000004608.
Usage exclusive: 9011888128
```
Disk image is still at set size of 11000004608.
```
sudo du -b /var/lib/lxd/storage-pools/btrfs/virtual-machines/v1/root.img
11000004608 /var/lib/lxd/storage-pools/btrfs/virtual-machines/v1/root.img
```
And the actual used blocks of the image are:
```
du -B1 root.img
10758160384 root.img
```
Indeed the total size of the volume is less than the quota:
```
du -B1
20480 ./templates
8192 ./config/systemd
4096 ./config/udev
4096 ./config/files
14815232 ./config
10773151744 .
```