LXD fails to startup after reboot

After reboot, LXD fails to start with following journal entries:

-- Reboot --
Okt 02 12:45:39 cirrus systemd[1]: Starting LXD - main daemon...
Okt 02 12:45:40 cirrus lxd[3852]: lvl=warn msg="CGroup memory swap accounting is    disabled, swap limits will be ignored." t=2018-10-02T12:45:
Okt 02 12:45:40 cirrus lxd[3852]: panic: log not found 
Okt 02 12:45:40 cirrus lxd[3852]: goroutine 1 [running]:
Okt 02 12:45:40 cirrus lxd[3852]: github.com/hashicorp/raft.NewRaft(0xc420032360, 0x11278e0, 0xc42030b060, 0x112fd20, 0xc42030ae60, 0x112c9e
Okt 02 12:45:40 cirrus lxd[3852]:         /build/lxd-0FDBXp/lxd-3.0.1/obj-x86_64-linux-gnu/src/github.com/hashicorp/raft/api.go:491 +0x11ba
Okt 02 12:45:40 cirrus lxd[3852]: github.com/lxc/lxd/lxd/cluster.raftInstanceInit(0xc42030a220, 0xc42030ad00, 0xc4202f7ea0, 0x40080000000000
Okt 02 12:45:40 cirrus lxd[3852]:         /build/lxd-0FDBXp/lxd-3.0.1/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/cluster/raft.go:190 +0
Okt 02 12:45:40 cirrus lxd[3852]: github.com/lxc/lxd/lxd/cluster.newRaft(0xc42030a220, 0xc4202f7ea0, 0x4008000000000000, 0x0, 0x0, 0x40d606)
Okt 02 12:45:40 cirrus lxd[3852]:         /build/lxd-0FDBXp/lxd-3.0.1/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/cluster/raft.go:71 +0x
Okt 02 12:45:40 cirrus lxd[3852]: github.com/lxc/lxd/lxd/cluster.(*Gateway).init(0xc42033a3c0, 0xc42033a3c0, 0xc420038e40)
Okt 02 12:45:40 cirrus lxd[3852]:         /build/lxd-0FDBXp/lxd-3.0.1/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/cluster/gateway.go:403
Okt 02 12:45:40 cirrus lxd[3852]: github.com/lxc/lxd/lxd/cluster.NewGateway(0xc42030a220, 0xc4202f7ea0, 0xc4202b76c0, 0x2, 0x2, 0xc42003a500
Okt 02 12:45:40 cirrus lxd[3852]:         /build/lxd-0FDBXp/lxd-3.0.1/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/cluster/gateway.go:54 
Okt 02 12:45:40 cirrus lxd[3852]: main.(*Daemon).init(0xc4202d8270, 0xc4202b78d8, 0x40d606)
Okt 02 12:45:40 cirrus lxd[3852]:         /build/lxd-0FDBXp/lxd-3.0.1/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/daemon.go:416 +0x4a8
Okt 02 12:45:40 cirrus lxd[3852]: main.(*Daemon).Init(0xc4202d8270, 0xc4202d8270, 0xc420038960)
Okt 02 12:45:40 cirrus lxd[3852]:         /build/lxd-0FDBXp/lxd-3.0.1/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/daemon.go:355 +0x2f
Okt 02 12:45:40 cirrus lxd[3852]: main.(*cmdDaemon).Run(0xc420031600, 0xc4202cc000, 0xc4202a6960, 0x0, 0x3, 0x0, 0x0)
Okt 02 12:45:40 cirrus lxd[3852]:         /build/lxd-0FDBXp/lxd-3.0.1/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/main_daemon.go:82 +0x3
Okt 02 12:45:40 cirrus lxd[3852]: main.(*cmdDaemon).Run-fm(0xc4202cc000, 0xc4202a6960, 0x0, 0x3, 0x0, 0x0)
Okt 02 12:45:40 cirrus lxd[3852]:         /build/lxd-0FDBXp/lxd-3.0.1/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/main_daemon.go:42 +0x5
Okt 02 12:45:40 cirrus lxd[3852]: github.com/spf13/cobra.(*Command).execute(0xc4202cc000, 0xc4200c0050, 0x3, 0x3, 0xc4202cc000, 0xc4200c0050
Okt 02 12:45:40 cirrus lxd[3852]:         /build/lxd-0FDBXp/lxd-3.0.1/obj-x86_64-linux-gnu/src/github.com/spf13/cobra/command.go:762 +0x468
Okt 02 12:45:40 cirrus lxd[3852]: github.com/spf13/cobra.(*Command).ExecuteC(0xc4202cc000, 0x0, 0xc4202d6500, 0xc4202d6500)
Okt 02 12:45:40 cirrus lxd[3852]:         /build/lxd-0FDBXp/lxd-3.0.1/obj-x86_64-linux-gnu/src/github.com/spf13/cobra/command.go:852 +0x30a
Okt 02 12:45:40 cirrus lxd[3852]: github.com/spf13/cobra.(*Command).Execute(0xc4202cc000, 0xc4202b7e08, 0x1)
Okt 02 12:45:40 cirrus lxd[3852]:         /build/lxd-0FDBXp/lxd-3.0.1/obj-x86_64-linux-gnu/src/github.com/spf13/cobra/command.go:800 +0x2b
Okt 02 12:45:40 cirrus lxd[3852]: main.main()
Okt 02 12:45:40 cirrus lxd[3852]:         /build/lxd-0FDBXp/lxd-3.0.1/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/main.go:159 +0xdca
Okt 02 12:45:40 cirrus systemd[1]: lxd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Okt 02 12:50:14 cirrus systemd[1]: lxd.service: Failed with result 'exit-code'.
Okt 02 12:50:14 cirrus systemd[1]: Stopped LXD - main daemon.

Last time LXD failed to startup, was right last week after my hard-drive ran full. I came across a thread somewhere in the internet (github?), which suggested to delete the db file /var/lib/lxd/database/global/logs.db. I did that, afterwards lxd started. After that there was no reboot, until now.

Which log is meant by panic: log not found?

Update: After repeating the step from last time (removing logs.db), LXD started again, also after reboot. What is logs.db about, how can have a look at it?

Update 2: I’m now back at the state from the moment my hard-disk went full. Which is bad, since I moved my container to a zfs pool since then.

Having a full disk is a critical issue that can make apps to misbehave. It gets even more crashy when you try that with a desktop environment or a Web browser.
There is not much to do when the disk is full. The application tries to write back the configuration and fails because there is no space left.
In such situations, the least that can be done, is to gracefully fail in some documented way.
I have not experienced this, and I do not know what should be happening.

What I know is that there is a backup of logs.db, in /var/lib/lxd/database/global.bak/logs.db.
If your /var/lib/lxd/database/global/logs.db has 0 size when you reboot, you can attempt to copy there the backup file, then restart the LXD service.

Is your disk getting full due to LXD? Do you use the dir storage backend?

Disk went full while I have copied my containers from dir storage to a zfs pool. I had to free space on my device to get Gnome starting again. And after that I deleted the logs.db and got lxd to start again. After that I moved my container successfully to a zfs pool.

I haven’t rebooted my machine after that, until this morning. LXD fails to start, I deleted logs.db again and the state was restored right before my disk went full. I think, it was restored from a snapshot.

This leaves me without my moved containers in the zfs pool …

All files in global.bak are from before the disk went full.

So my problem would be solved If it would be possible to reintegrate the container in the zfs storage again.