I don’t see any errors in systemd journal on the server, nor is there anything (at all) in incusd.log. the containers lxc.log only show what is visible above - nothing - and console.log appears to show the console when I last had this container running.
None of my changes in apts history.log or /etc/ lead me to a cause.
Any suggestions on what might have caused this, or how to investigate it further?
Thanks for your help @stgraber , sorry it took a while to respond.
I can confirm its btrfs type
root@s26001:~# incus storage list
+---------+--------+-------------+---------+---------+
| NAME | DRIVER | DESCRIPTION | USED BY | STATE |
+---------+--------+-------------+---------+---------+
| default | btrfs | | 4 | CREATED |
+---------+--------+-------------+---------+---------+
hopefully these are of use in debugging
root@s26001:~# df -h /var/lib/incus/containers/llama-cpp-server
Filesystem Size Used Avail Use% Mounted on
- 1.6T 62G 1.5T 4% /var/lib/incus/storage-pools/default/containers/llama-cpp-server
root@s26001:~# mount |grep 'sda.*incus'
/dev/sda4 on /incusvol type btrfs (rw,relatime,ssd,discard=async,space_cache=v2,user_subvol_rm_allowed,subvolid=5,subvol=/)
/dev/sda4 on /var/lib/incus/storage-pools/default type btrfs (rw,relatime,ssd,discard=async,space_cache=v2,user_subvol_rm_allowed,subvolid=5,subvol=/)
/dev/sda4 on /var/lib/incus/devices/OperationsCenterAppliance/config.mount type btrfs (ro,relatime,ssd,discard=async,space_cache=v2,user_subvol_rm_allowed,subvolid=256,subvol=/virtual-machines/OperationsCenterAppliance)
root@s26001:~# ls -lha /var/lib/incus/storage-pools/default/containers/llama-cpp-server
total 12K
d--x------ 1 1000000 root 78 Aug 24 19:30 .
drwx--x--x 1 root root 32 Aug 21 17:00 ..
-r-------- 1 root root 6.6K Aug 22 21:01 backup.yaml
-rw-r--r-- 1 root root 529 Aug 20 15:28 metadata.yaml
drwxr-xr-x 1 root root 216 Aug 21 17:17 rootfs
drwxr-xr-x 1 root root 42 Aug 20 15:28 templates
root@s26001:~# incus config device override llama-cpp-server root size=33GB
Error: Failed to update device "root": Failed to run: btrfs qgroup limit 33000000000 0/258 /var/lib/incus/storage-pools/default/containers/llama-cpp-server: exit status 1 (ERROR: unable to limit requested quota group: Disk quota exceeded)
root@s26001:~# incus config device override llama-cpp-server root size.state=33GB
Error: Failed to update device "root": Failed to run: btrfs qgroup limit 30000000000 0/258 /var/lib/incus/storage-pools/default/containers/llama-cpp-server: exit status 1 (ERROR: unable to limit requested quota group: Disk quota exceeded)
I had a look in system journal but don’t see any messages logged at that time.
In case the issue with btrfs was kernel version specific I rebooted from 6.12.101+deb13-amd64 into 6.12.73+deb13-amd64 but the disk quota issue persists - for root and incus
root@s26001:~# incus config device override llama-cpp-server root size.state=33GB
Error: Failed to write backup file: Failed to create file "/var/lib/incus/containers/llama-cpp-server/backup.yaml": openat backup.yaml: disk quota exceeded
root@s26001:~# mkdir /var/lib/incus/containers/llama-cpp-server/credentials
mkdir: cannot create directory ‘/var/lib/incus/containers/llama-cpp-server/credentials’: Disk quota exceeded
Hmm, okay, then the next obvious option would be to clear up some space in there to get the quota errors to stop so you can then update the quota or at least start it back up.
You can either look for anything obvious to clear directly through /var/lib/incus/containers/llama-cpp-server/rootfs/ or you can use incus file mount to mount the container’s filesystem on any system that can interact with Incus.
Good candidates for quickly clearing space would be /var/cache/apt/archives/ or stuff under /var/log/
I found a 5GB cache directory in roots home and that freed up space which allowed me to start up. The rootfs directory reports as being 29GB (probably 30 if i had used du --si).