Cant start virtual machine, disk quota exceeded. Cant grow disk quota or attach new storage pool either

LXD
#1

I am using debian 11 as the host and have installed lxd via snap using the btrfs storage type since that was default when I initially ran lxd init.

When I try to run an ubuntu virtual machine, I get the following error:

$ lxc start ubuntu
Error: open /var/snap/lxd/common/lxd/virtual-machines/ubuntu/config/server.crt: disk quota exceeded
Try `lxc info --show-log ubuntu` for more info

when I run more info I get:

Name: ubuntu
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: REDACTED
Last Used: REDACTED
Error: open /var/snap/lxd/common/lxd/logs/ubuntu/qemu.log: no such file or director

when I try to edit the disk size from 15GB to something larger, I get the following error:

$ lxc config edit ubuntu
Config parsing error: Failed to update device "root": Failed resizing disk image "/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/ubuntu/root.img" to size 20000006144: Failed to create sparse file /var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/ubuntu/root.img: truncate /var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/ubuntu/root.img: disk quota exceeded
Press enter to open the editor again or ctrl+c to abort change

Also note that just opening the config without making any changes gives me an error:

$ lxc config edit ubuntu
Config parsing error: Failed to write backup file: Failed to create file "/var/snap/lxd/common/lxd/virtual-machines/ubuntu/backup.yaml": open /var/snap/lxd/common/lxd/virtual-machines/ubuntu/backup.yaml: disk quota exceeded
Press enter to open the editor again or ctrl+c to abort change

I dont really understand how all this works, but after skimming the documentation again to try and find info on how to increase the disk size, I tried attaching a new storage volume with:

$ lxc storage create tst3 btrfs
$ lxc storage volume create tst3 tstblockvol1 --type=block
$ lxc storage volume attach tst3 tstblockvol1 ubuntu
Error: Failed to write backup file: Failed to create file "/var/snap/lxd/common/lxd/virtual-machines/ubuntu/backup.yaml": open /var/snap/lxd/common/lxd/virtual-machines/ubuntu/backup.yaml: disk quota exceeded

I’m very new to lxd, and don’t work in a tech field and am pretty out of my depth, I just wanted something more flexible / powerful then virtualbox and I could not get virt machine manager working the way I wanted it to. so any advice / suggestions would be appreciated.

#2

Please see the 4th bullet point here

About setting the size.state setting on the root disk when using btrfs (or even better don’t use btrfs for VMS).

#3

Also check the storage pool size and check you’ve not just run out of disk space in the pool

#4

Please see the 4th bullet point here

I dont understand any of that. I dont work in a technology feild. I see that it says " please ensure that the instance root disk’s size.state property is set to 2x the size of the root disk’s size to allow all blocks in the disk image file to be rewritten without reaching the qgroup quota.", but I dont know what to do with that information. there is no “size.state” entry in the config of that particular virtual machine, and I am not sure what to set it to or how to change it.

Also check the storage pool size and check you’ve not just run out of disk space in the pool

I have not. or at least, when I run ‘lxc storage edit default’ and jack up the size of the disk it does nothing to change the error message.

or even better don’t use btrfs for VMS).

Apologies for using the default configuration. If its not recommended to use the default configuration, I would suggest changing the default configuration.

#5

In LXD a VM is made up of 2 volumes; a small state volume (that contains config and metadata about the VM), and the data volume (that contains the VM’s root disk).

By default in LXD, BTRFS storage pools don’t have per filesystem volume size limits, so the state volume won’t be size limited. The VM root volume itself will be an image file sized to the pool’s volume.size setting or if not set will default to 10GiB.

If you grow the VM’s root disk size after creation using:

lxc config device set <instance> root size=20GiB

Then this will resize the root image file to the specified size, and enable BTRFS quotas with a quota set at the specified size + the state volume’s default size (which is 100MiB).

However because of the nature of BTRFS’ quota tracking (as explained in that 4th bullet point) it is possible for the BTRFS quota to be exceeded due to the way that BTRFS tracks the blocks that are changing inside the VM’s root image file. The absolute worst case scenario is that it could actually keep track of up to 2x the allowed quota when using a VM, so we encourage setting the state volume size to 2x the specified root volume size to be safe.

This can be done by doing:

lxc config device set <instance> root size.state=40GiB

This will cause the BTRFS quota to be set at 20+40GiB to allow quota to track block changes in the VM’s root disk.

This is all rather complicated, and this is why we actually suggest not using BTRFS for VMs.

On your point regarding LXD suggesting BTRFS for the default. The default pool type is actually ZFS, but its only suggested if your system has it available. The “next best” (although we’ve seen now that “best” is somewhat nuanced based on your workload and preferences) is suggested as BTRFS, because it is the next most flexible and efficient for containers (which LXD started out only supporting).

In both cases LXD will by default create BTRFS and ZFS pools on a fixed size loop file. This is to allow users to get up and running quickly, without having to provide a dedicated disk or partition for their instances. This works great for testing or development, however stacking filesystems ontop of the host’s filesystem is not as efficient, so for production workloads we recommend not using loop-backed pools, see:

Please can I see the output of the follow commands to better ascertain the state of your system:

  • lxc storage ls
  • lxc storage show default
  • lxc config show <instance> --expanded

Thanks

#6

Thank you for the explanation, but It still does not make a whole lot of sense but I dont think this is the time or place to try to educate me on what a volume is. I think i got the jist of it though.

Please can I see the output of the follow commands to better ascertain the state of your system:

Its difficult since i would need to redact anything that could identify me and there are to many options that I dont recognize.

Regardless, ran the command you suggested

lxc config device set ubuntu root size.state=40GiB

which did not work, so I just tried upping the size to 60, and now at least I can edit the config without getting errors. the size parts now read:

size: 20GB
size.state: 80GiB

but when I attempt to start the VM, it starts but wont allow me to connect with lxc exec ubuntu bash, just spits out Error: LXD VM agent isn't currently running. googling for that error spits out only the lxd github repo, and no troubleshooting help.

Attempting to use the spice window thing just shows the lxd logo forever. I also cant stop the VM unless I use the -f flag. Jacking up the size to 80 did not help either.

#7

Which image did you use to create the VM?

#8

A volume is just a real or virtual hard disk