Error: Failed to start dqlite server: raft_start()

Hi, I have installed LXD from source for some time, unfortunately today at the restart with the command:

LD_LIBRARY_PATH=$LD_LIBRARY_PATH $GOPATH/bin/lxd --verbose --debug

it gives me this error:

DBUG[02-25|18:30:23] Connecting to a local LXD over a Unix socket 
DBUG[02-25|18:30:23] Sending request to LXD                   method=GET url=http://unix.socket/1.0 etag=
INFO[02-25|18:30:23] LXD 3.20 is starting in normal mode      path=/var/lib/lxd
INFO[02-25|18:30:23] Kernel uid/gid map: 
INFO[02-25|18:30:23]  - u 0 0 4294967295 
INFO[02-25|18:30:23]  - g 0 0 4294967295 
INFO[02-25|18:30:23] Configured LXD uid/gid map: 
INFO[02-25|18:30:23]  - u 0 1000000 65536 
INFO[02-25|18:30:23]  - g 0 1000000 65536 
INFO[02-25|18:30:23] Kernel features: 
INFO[02-25|18:30:23]  - netnsid-based network retrieval: no 
INFO[02-25|18:30:23]  - uevent injection: yes 
INFO[02-25|18:30:23]  - seccomp listener: no 
INFO[02-25|18:30:23]  - seccomp listener continue syscalls: no 
INFO[02-25|18:30:23]  - unprivileged file capabilities: yes 
INFO[02-25|18:30:23]  - cgroup layout: hybrid 
WARN[02-25|18:30:23]  - Couldn't find the CGroup memory swap accounting, swap limits will be ignored 
INFO[02-25|18:30:23]  - shiftfs support: no 
INFO[02-25|18:30:23] Initializing local database 
DBUG[02-25|18:30:23] Initializing database gateway 
DBUG[02-25|18:30:23] Start database node                      id=1 address= role=voter
EROR[02-25|18:30:23] Failed to start the daemon: Failed to start dqlite server: raft_start(): io: load closed segment 0000000000002219-0000000000002266: found 47 entries (expected 48) 
INFO[02-25|18:30:23] Starting shutdown sequence 
DBUG[02-25|18:30:23] Not unmounting temporary filesystems (containers are still running) 
Error: Failed to start dqlite server: raft_start(): io: load closed segment 0000000000002219-0000000000002266: found 47 entries (expected 48)

It would seem linked to the dqlite database. Unfortunately I’m not an expert and ask for help to recover the db (if possible). It has happened to me before and then I solved by reinstalling everything, but this time I would try to avoid it.

@freeekanayaka should be able to help. I know he ended up reverting a go-dqlite change not long ago that had a similar behavior. Are you sure your copy of go-dqlite is up to date?

Make a backup of /var/lib/lxd/database then run

rm /var/lib/lxd/database/0000000000002219-0000000000002266

and restart LXD. Assuming it starts ok, just check that everything is fine (e.g. your containers and config is all there).

As @stgraber says we had a bug in go-dqlite, although it shouldn’t be related to your issue. Did you experience any crash/error or other particular events lately?

I have followed your instructions and everything seems to be working again. Thank you.

As for the question, there were no errors or crash events, but I do not know if the problem is related to the fact that I manually start lxd but I do not stop it when I turn off the PC (and I have been doing this for several months and this one, as said, it’s the second time I have this problem in the absence of critical events).

How do you “manually start” LXD and how do you turn off the PC exactly? As long as the LXD daemon receives a SIGTERM or SIGINT signal and has time to gracefully shutdown, everything should be fine. If that’s not the case you might occasionally hit a bug like this. I highly recommend using an init system to start and stop LXD.

I execute on a terminal this startup script:


export GOPATH=${HOME}/go
export LD_LIBRARY_PATH="${GOPATH}/deps/sqlite/.libs/:${GOPATH}/deps/dqlite/.libs/:${GOPATH}/deps/raft/.libs:${GOPATH}/deps/libco/:${LD_LIBRARY_PATH}"
export PATH="${GOPATH}/bin:${PATH}"

# LXD: for build
export CGO_CFLAGS="${CGO_CFLAGS} -I${GOPATH}/deps/sqlite/ -I${GOPATH}/deps/dqlite/include/ -I${GOPATH}/deps/raft/include/ -I${GOPATH}/deps/libco/"
export CGO_LDFLAGS="${CGO_LDFLAGS} -L${GOPATH}/deps/sqlite/.libs/ -L${GOPATH}/deps/dqlite/.libs/ -L${GOPATH}/deps/raft/.libs -L${GOPATH}/deps/libco/"

# LXD: check lxd
sudo lsof /var/lib/lxd/unix.socket 1>/dev/null 2>&1
if test $? -ne 0 ; then
	nohup sudo -E LD_LIBRARY_PATH=$LD_LIBRARY_PATH $GOPATH/bin/lxd --group sudo &

I use lxd for development and not for the provision of services, so I turn on and off my debian 10 workstation daily through gnome commands (I would like to put it under systemd but so far I have not done it)