aaron
(Aaron Whitehouse)
January 1, 2021, 8:52pm
1
Hello all,
I use LXD casually and on this machine have been using it for pihole and, just recently, Jellyfin. Stupidly, while I added in devices from the host for my Jellyfin media etc, I did not notice that Jellyfin was filling up my small ZFS loop file (5GB) with transcodes.
I believe that this storage volume is completely full and now nothing relating to LXD is working enough for me to recover the situation, so I am keen for any thoughts.
I should be able to recreate these two environments pretty quickly, but thought I would post here in case there is anything helpful for improving the resilience of LXD in these situations.
Examples:
$ lxc start jellyfin
$ lxc stop jellyfin
$ lxc exec jellyfin bash
$ lxc storage list`
just hang forever.
I tried to add an extra GB to my ZFS loopback following this guide:
but it hangs forever on:
$ sudo zpool set autoexpand=on lxd
Any logs etc that would be helpful, please let me know.
System info:
I am running Ubuntu:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic
$ which lxd
/snap/bin/lxd
$ which lxc
/snap/bin/lxc
LXD snap version:
installed: 4.9 (18772) 74MB
tomp
(Thomas Parrott)
January 4, 2021, 9:00am
2
Does LXD fail to start?
Can you enable debug logging and show the log output:
sudo snap set lxd daemon.debug=true; sudo systemctl reload snap.lxd.daemon
sudo tail -f /var/snap/lxd/common/lxd/logs/lxd.log
aaron
(Aaron Whitehouse)
January 4, 2021, 9:46pm
3
Thank you for the prompt and helpful reply, @tomp – the output of your suggested commands is below:
$ sudo tail -f /var/snap/lxd/common/lxd/logs/lxd.log
t=2021-01-04T21:45:40+0000 lvl=info msg="REST API daemon:"
t=2021-01-04T21:45:40+0000 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
t=2021-01-04T21:45:40+0000 lvl=info msg="Initializing global database"
t=2021-01-04T21:45:40+0000 lvl=dbug msg="Dqlite: attempt 0: server 1: connected"
t=2021-01-04T21:45:40+0000 lvl=dbug msg="Firewall detected \"nftables\" incompatibility: Kernel version does not meet minimum requirement of 5"
t=2021-01-04T21:45:40+0000 lvl=dbug msg="Firewall xtables detected iptables is in use"
t=2021-01-04T21:45:40+0000 lvl=info msg="Firewall loaded driver \"xtables\""
t=2021-01-04T21:45:40+0000 lvl=info msg="Initializing storage pools"
t=2021-01-04T21:45:40+0000 lvl=dbug msg="Initializing and checking storage pool \"default\""
t=2021-01-04T21:45:40+0000 lvl=dbug msg="Mount started" driver=zfs pool=default
tomp
(Thomas Parrott)
January 5, 2021, 8:56am
4
So it looks like the activation of the ZFS pool is hanging preventing LXD from starting.
Are you seeing anything in journalctl -f
or sudo dmesg
at the same time from ZFS hinting at the problem?