Recovering from completely full ZFS storage loop file

Hello all,

I use LXD casually and on this machine have been using it for pihole and, just recently, Jellyfin. Stupidly, while I added in devices from the host for my Jellyfin media etc, I did not notice that Jellyfin was filling up my small ZFS loop file (5GB) with transcodes.

I believe that this storage volume is completely full and now nothing relating to LXD is working enough for me to recover the situation, so I am keen for any thoughts.

I should be able to recreate these two environments pretty quickly, but thought I would post here in case there is anything helpful for improving the resilience of LXD in these situations.

Examples:

$ lxc start jellyfin
$ lxc stop jellyfin
$ lxc exec jellyfin bash
$ lxc storage list`

just hang forever.

I tried to add an extra GB to my ZFS loopback following this guide:


but it hangs forever on:
$ sudo zpool set autoexpand=on lxd

Any logs etc that would be helpful, please let me know.

System info:
I am running Ubuntu:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.5 LTS
Release:	18.04
Codename:	bionic

$ which lxd
/snap/bin/lxd

$ which lxc
/snap/bin/lxc

LXD snap version:

installed:          4.9                    (18772) 74MB 

Does LXD fail to start?

Can you enable debug logging and show the log output:

sudo snap set lxd daemon.debug=true; sudo systemctl reload snap.lxd.daemon
sudo tail -f /var/snap/lxd/common/lxd/logs/lxd.log

Thank you for the prompt and helpful reply, @tomp – the output of your suggested commands is below:

$ sudo tail -f /var/snap/lxd/common/lxd/logs/lxd.log
t=2021-01-04T21:45:40+0000 lvl=info msg="REST API daemon:" 
t=2021-01-04T21:45:40+0000 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
t=2021-01-04T21:45:40+0000 lvl=info msg="Initializing global database" 
t=2021-01-04T21:45:40+0000 lvl=dbug msg="Dqlite: attempt 0: server 1: connected" 
t=2021-01-04T21:45:40+0000 lvl=dbug msg="Firewall detected \"nftables\" incompatibility: Kernel version does not meet minimum requirement of 5" 
t=2021-01-04T21:45:40+0000 lvl=dbug msg="Firewall xtables detected iptables is in use" 
t=2021-01-04T21:45:40+0000 lvl=info msg="Firewall loaded driver \"xtables\"" 
t=2021-01-04T21:45:40+0000 lvl=info msg="Initializing storage pools" 
t=2021-01-04T21:45:40+0000 lvl=dbug msg="Initializing and checking storage pool \"default\"" 
t=2021-01-04T21:45:40+0000 lvl=dbug msg="Mount started" driver=zfs pool=default

So it looks like the activation of the ZFS pool is hanging preventing LXD from starting.

Are you seeing anything in journalctl -f or sudo dmesg at the same time from ZFS hinting at the problem?