Error: Failed to start dqlite server: raft_start(): io: closed segment 0000000000029825-0000000000030604 is past last snapshot snapshot-1-29696-3876743636

Hello,
Help me, please,
snap.lxd.daemon.service permanently fails:

Jul 20 09:56:12 carambola systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Jul 20 09:56:12 carambola systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Jul 20 09:56:12 carambola systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 1.
Jul 20 09:56:12 carambola systemd[1]: Stopped Service for snap application lxd.daemon.
Jul 20 09:56:12 carambola systemd[1]: Started Service for snap application lxd.daemon.
Jul 20 09:56:12 carambola lxd.daemon[4030]: => Preparing the system (21039)
Jul 20 09:56:12 carambola lxd.daemon[4030]: ==> Loading snap configuration
Jul 20 09:56:12 carambola lxd.daemon[4030]: ==> Setting up mntns symlink (mnt:[4026533085])
Jul 20 09:56:12 carambola lxd.daemon[4030]: ==> Setting up kmod wrapper
Jul 20 09:56:12 carambola lxd.daemon[4030]: ==> Preparing /boot
Jul 20 09:56:12 carambola lxd.daemon[4030]: ==> Preparing a clean copy of /run
Jul 20 09:56:12 carambola lxd.daemon[4030]: ==> Preparing /run/bin
Jul 20 09:56:12 carambola lxd.daemon[4030]: ==> Preparing a clean copy of /etc
Jul 20 09:56:12 carambola lxd.daemon[4030]: ==> Preparing a clean copy of /usr/share/misc
Jul 20 09:56:12 carambola lxd.daemon[4030]: ==> Setting up ceph configuration
Jul 20 09:56:12 carambola lxd.daemon[4030]: ==> Setting up LVM configuration
Jul 20 09:56:12 carambola lxd.daemon[4030]: ==> Rotating logs
Jul 20 09:56:12 carambola lxd.daemon[4030]: ==> Setting up ZFS (0.8)
Jul 20 09:56:12 carambola lxd.daemon[4030]: ==> Escaping the systemd cgroups
Jul 20 09:56:12 carambola lxd.daemon[4030]: ====> Detected cgroup V1
Jul 20 09:56:12 carambola lxd.daemon[4030]: ==> Escaping the systemd process resource limits
Jul 20 09:56:12 carambola lxd.daemon[4030]: ==> Disabling shiftfs on this kernel (auto)
Jul 20 09:56:12 carambola lxd.daemon[2301]: Closed liblxcfs.so
Jul 20 09:56:12 carambola lxd.daemon[2301]: Running destructor lxcfs_exit
Jul 20 09:56:12 carambola lxd.daemon[2301]: Running constructor lxcfs_init to reload liblxcfs
Jul 20 09:56:12 carambola lxd.daemon[2301]: mount namespace: 5
Jul 20 09:56:12 carambola lxd.daemon[2301]: hierarchies:
Jul 20 09:56:12 carambola lxd.daemon[2301]:   0: fd:   6:
Jul 20 09:56:12 carambola lxd.daemon[2301]:   1: fd:   7: name=systemd
Jul 20 09:56:12 carambola lxd.daemon[2301]:   2: fd:   8: cpuset
Jul 20 09:56:12 carambola lxd.daemon[2301]:   3: fd:   9: perf_event
Jul 20 09:56:12 carambola lxd.daemon[2301]:   4: fd:  10: devices
Jul 20 09:56:12 carambola lxd.daemon[2301]:   5: fd:  11: cpu,cpuacct
Jul 20 09:56:12 carambola lxd.daemon[2301]:   6: fd:  12: pids
Jul 20 09:56:12 carambola lxd.daemon[2301]:   7: fd:  13: freezer
Jul 20 09:56:12 carambola lxd.daemon[2301]:   8: fd:  14: hugetlb
Jul 20 09:56:12 carambola lxd.daemon[2301]:   9: fd:  15: memory
Jul 20 09:56:12 carambola lxd.daemon[2301]:  10: fd:  16: blkio
Jul 20 09:56:12 carambola lxd.daemon[2301]:  11: fd:  17: net_cls,net_prio
Jul 20 09:56:12 carambola lxd.daemon[2301]:  12: fd:  19: rdma
Jul 20 09:56:12 carambola lxd.daemon[2301]: Kernel supports pidfds
Jul 20 09:56:12 carambola lxd.daemon[2301]: Kernel supports swap accounting
Jul 20 09:56:12 carambola lxd.daemon[2301]: api_extensions:
Jul 20 09:56:12 carambola lxd.daemon[2301]: - cgroups
Jul 20 09:56:12 carambola lxd.daemon[2301]: - sys_cpu_online
Jul 20 09:56:12 carambola lxd.daemon[2301]: - proc_cpuinfo
Jul 20 09:56:12 carambola lxd.daemon[2301]: - proc_diskstats
Jul 20 09:56:12 carambola lxd.daemon[2301]: - proc_loadavg
Jul 20 09:56:12 carambola lxd.daemon[2301]: - proc_meminfo
Jul 20 09:56:12 carambola lxd.daemon[2301]: - proc_stat
Jul 20 09:56:12 carambola lxd.daemon[2301]: - proc_swaps
Jul 20 09:56:12 carambola lxd.daemon[2301]: - proc_uptime
Jul 20 09:56:12 carambola lxd.daemon[2301]: - shared_pidns
Jul 20 09:56:12 carambola lxd.daemon[2301]: - cpuview_daemon
Jul 20 09:56:12 carambola lxd.daemon[2301]: - loadavg_daemon
Jul 20 09:56:12 carambola lxd.daemon[2301]: - pidfds
Jul 20 09:56:12 carambola lxd.daemon[2301]: Reloaded LXCFS
Jul 20 09:56:12 carambola lxd.daemon[4030]: => Re-using existing LXCFS
Jul 20 09:56:12 carambola lxd.daemon[4030]: => Starting LXD
Jul 20 09:56:12 carambola lxd.daemon[4166]: t=2021-07-20T09:56:12+0300 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
Jul 20 09:56:13 carambola lxd.daemon[4166]: t=2021-07-20T09:56:13+0300 lvl=eror msg="Failed to start the daemon: Failed to start dqlite server: raft_start(): io: closed segment 0000000000029825-000000000>
Jul 20 09:56:13 carambola lxd.daemon[4166]: Error: Failed to start dqlite server: raft_start(): io: closed segment 0000000000029825-0000000000030604 is past last snapshot snapshot-1-29696-3876743636
Jul 20 09:56:13 carambola lxd.daemon[4030]: => LXD failed to start
Jul 20 09:56:13 carambola systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Jul 20 09:56:13 carambola systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Jul 20 09:56:14 carambola systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 2.
Jul 20 09:56:14 carambola systemd[1]: Stopped Service for snap application lxd.daemon.
Jul 20 09:56:14 carambola systemd[1]: Started Service for snap application lxd.daemon.

What can be wrong with it?

@mbordere please can you take a look at this one? Thanks

hi @512YB , Can you paste the output of ls -alh /var/snap/lxd/common/lxd/database/global please?

Do you still have enough free disk space?