Here is the lxd log on the node that went off (host and ip addresses have been redacted):
journalctl -u snap.lxd.daemon -n 100
-- Logs begin at Fri 2021-03-26 12:26:54 UTC, end at Wed 2021-11-17 12:59:13 UTC. --
Nov 16 11:03:24 server1 lxd.daemon[2148742]: github.com/lxc/lxd/lxd/storage/drivers.(*btrfs).getSubvolumes(0xc03422c1e0, {0xc02495d110, 0x62})
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /build/lxd/parts/lxd/src/lxd/storage/drivers/driver_btrfs_utils.go:62 +0xee
Nov 16 11:03:24 server1 lxd.daemon[2148742]: github.com/lxc/lxd/lxd/storage/drivers.(*btrfs).deleteSubvolume(0xc03422c1e0, {0xc02495d110, 0x62}, 0x1)
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /build/lxd/parts/lxd/src/lxd/storage/drivers/driver_btrfs_utils.go:162 +0xa7
Nov 16 11:03:24 server1 lxd.daemon[2148742]: github.com/lxc/lxd/lxd/storage/drivers.(*btrfs).DeleteVolumeSnapshot(0xc03422c1e0, {{0xc0c4619960, 0x20}, {0xc0e01e2ec5, 0x5}, 0xc071f83c80, {0x18a5d1b, 0xa}, {0x18a617b, 0xa}, ...}, ...)
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /build/lxd/parts/lxd/src/lxd/storage/drivers/driver_btrfs_volumes.go:1319 +0xb1
Nov 16 11:03:24 server1 lxd.daemon[2148742]: github.com/lxc/lxd/lxd/storage.(*lxdBackend).DeleteInstanceSnapshot(0xc04f45fd40, {0x1bfb578, 0xc05e6de6e0}, 0xc0c5b62130)
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /build/lxd/parts/lxd/src/lxd/storage/backend_lxd.go:2338 +0x7c3
Nov 16 11:03:24 server1 lxd.daemon[2148742]: github.com/lxc/lxd/lxd/instance/drivers.(*lxc).Delete(0xc05e6de6e0, 0x1)
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /build/lxd/parts/lxd/src/lxd/instance/drivers/driver_lxc.go:3635 +0x484
Nov 16 11:03:24 server1 lxd.daemon[2148742]: main.pruneExpiredContainerSnapshots({0xc0a2a20500, 0x77ba6a}, 0x1bea470, {0xc024265100, 0x8, 0x10100000174bd80})
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /build/lxd/parts/lxd/src/lxd/instance.go:612 +0x88
Nov 16 11:03:24 server1 lxd.daemon[2148742]: main.pruneExpiredContainerSnapshotsTask.func1.1(0x41a634)
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /build/lxd/parts/lxd/src/lxd/instance.go:575 +0x31
Nov 16 11:03:24 server1 lxd.daemon[2148742]: github.com/lxc/lxd/lxd/operations.(*Operation).Run.func1(0xc049c71560, 0xc020525710)
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /build/lxd/parts/lxd/src/lxd/operations/operations.go:270 +0x42
Nov 16 11:03:24 server1 lxd.daemon[2148742]: created by github.com/lxc/lxd/lxd/operations.(*Operation).Run
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /build/lxd/parts/lxd/src/lxd/operations/operations.go:269 +0x128
Nov 16 11:03:24 server1 lxd.daemon[2148742]: goroutine 316834 [syscall, 346 minutes]:
Nov 16 11:03:24 server1 lxd.daemon[2148742]: syscall.Syscall6(0x106, 0xffffffffffffff9c, 0xc164b6c820, 0xc0a9d7ad38, 0x100, 0x0, 0x0)
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /snap/go/8489/src/syscall/asm_linux_amd64.s:43 +0x5
Nov 16 11:03:24 server1 lxd.daemon[2148742]: syscall.fstatat(0x0, {0xc164b6c750, 0xca}, 0xc0a9d7ad38, 0xca)
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /snap/go/8489/src/syscall/zsyscall_linux_amd64.go:1441 +0x10f
Nov 16 11:03:24 server1 lxd.daemon[2148742]: syscall.Lstat(...)
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /snap/go/8489/src/syscall/syscall_linux_amd64.go:74
Nov 16 11:03:24 server1 lxd.daemon[2148742]: os.lstatNolog.func1(...)
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /snap/go/8489/src/os/stat_unix.go:46
Nov 16 11:03:24 server1 lxd.daemon[2148742]: os.ignoringEINTR(...)
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /snap/go/8489/src/os/file_posix.go:246
Nov 16 11:03:24 server1 lxd.daemon[2148742]: os.lstatNolog({0xc164b6c750, 0x3})
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /snap/go/8489/src/os/stat_unix.go:45 +0x5b
Nov 16 11:03:24 server1 lxd.daemon[2148742]: os.Lstat({0xc164b6c750, 0xca})
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /snap/go/8489/src/os/stat.go:22 +0x34
Nov 16 11:03:24 server1 lxd.daemon[2148742]: path/filepath.walk({0xc002166360, 0x89}, {0x1b9f358, 0xc00eec2410}, 0xc002f2d430)
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /snap/go/8489/src/path/filepath/path.go:436 +0x1db
Nov 16 11:03:24 server1 lxd.daemon[2148742]: path/filepath.walk({0xc003600900, 0x7e}, {0x1b9f358, 0xc00eec2340}, 0xc002f2d430)
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /snap/go/8489/src/path/filepath/path.go:442 +0x28f
Nov 16 11:03:24 server1 lxd.daemon[2148742]: path/filepath.walk({0xc003600500, 0x78}, {0x1b9f358, 0xc00eec2270}, 0xc002f2d430)
Nov 16 11:03:24 server1 lxd.daemon[2148742]: /snap/go/8489/src/path/filepath/path.go:442 +0x28f
Nov 16 13:41:08 server1 lxd.daemon[2148194]: => LXD failed with return code 2
Nov 16 13:41:08 server1 systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Nov 16 13:41:08 server1 systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Nov 16 13:41:08 server1 systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 1.
Nov 16 13:41:08 server1 systemd[1]: Stopped Service for snap application lxd.daemon.
Nov 16 13:41:08 server1 systemd[1]: Started Service for snap application lxd.daemon.
Nov 16 13:41:10 server1 lxd.daemon[2373891]: => Preparing the system (21858)
Nov 16 13:41:10 server1 lxd.daemon[2373891]: ==> Loading snap configuration
Nov 16 13:41:10 server1 lxd.daemon[2373891]: ==> Setting up mntns symlink (mnt:[4026533313])
Nov 16 13:41:10 server1 lxd.daemon[2373891]: ==> Setting up kmod wrapper
Nov 16 13:41:10 server1 lxd.daemon[2373891]: ==> Preparing /boot
Nov 16 13:41:10 server1 lxd.daemon[2373891]: ==> Preparing a clean copy of /run
Nov 16 13:41:10 server1 lxd.daemon[2373891]: ==> Preparing /run/bin
Nov 16 13:41:10 server1 lxd.daemon[2373891]: ==> Preparing a clean copy of /etc
Nov 16 13:41:11 server1 lxd.daemon[2373891]: ==> Preparing a clean copy of /usr/share/misc
Nov 16 13:41:11 server1 lxd.daemon[2373891]: ==> Setting up ceph configuration
Nov 16 13:41:11 server1 lxd.daemon[2373891]: ==> Setting up LVM configuration
Nov 16 13:41:11 server1 lxd.daemon[2373891]: ==> Rotating logs
Nov 16 13:41:11 server1 lxd.daemon[2373891]: ==> Setting up ZFS (0.8)
Nov 16 13:41:11 server1 lxd.daemon[2373891]: ==> Escaping the systemd cgroups
Nov 16 13:41:11 server1 lxd.daemon[2373891]: ====> Detected cgroup V1
Nov 16 13:41:11 server1 lxd.daemon[2373891]: ==> Escaping the systemd process resource limits
Nov 16 13:41:11 server1 lxd.daemon[2373891]: ==> Disabling shiftfs on this kernel (auto)
Nov 16 13:41:12 server1 lxd.daemon[2373891]: => Re-using existing LXCFS
Nov 16 13:41:12 server1 lxd.daemon[2373891]: ==> Cleaning up existing LXCFS namespace
Nov 16 13:41:12 server1 lxd.daemon[2373891]: => Starting LXD
Nov 16 13:41:13 server1 lxd.daemon[2374581]: t=2021-11-16T13:41:13+0000 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
Nov 16 13:41:13 server1 lxd.daemon[2374581]: t=2021-11-16T13:41:13+0000 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
Nov 16 13:41:15 server1 lxd.daemon[2374581]: t=2021-11-16T13:41:15+0000 lvl=warn msg="Dqlite: attempt 1: server xxx.xx.xx.xx6:8443: no known leader"
Nov 16 13:41:18 server1 lxd.daemon[2374581]: t=2021-11-16T13:41:18+0000 lvl=warn msg="Failed to initialize fanotify, falling back on fsnotify" err="Failed to initialize fanotify: invalid argument"
Nov 16 13:41:20 server1 lxd.daemon[2373891]: => LXD is ready
Nov 16 13:41:27 server1 lxd.daemon[2374581]: t=2021-11-16T13:41:27+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T11:03:13+0000 raftID=4
Nov 16 15:37:02 server1 lxd.daemon[2374581]: 2021/11/16 15:37:02 http: TLS handshake error from 192.241.206.121:39970: tls: client used the legacy version field to negotiate TLS 1.3
Nov 16 16:22:40 server1 lxd.daemon[2374581]: 2021/11/16 16:22:40 http: TLS handshake error from 192.241.205.109:52792: tls: client offered only unsupported versions: [302 301]
Nov 16 17:05:41 server1 lxd.daemon[2374581]: t=2021-11-16T17:05:41+0000 lvl=warn msg="Failed to get events from member" address=xxx.xx.xx.xx7:8443 err="Unable to connect to: xxx.xx.xx.xx7:8443"
Nov 16 17:05:42 server1 lxd.daemon[2374581]: t=2021-11-16T17:05:42+0000 lvl=warn msg="Failed to get events from member" address=xxx.xx.xx.xx7:8443 err="Unable to connect to: xxx.xx.xx.xx7:8443"
Nov 16 17:05:43 server1 lxd.daemon[2374581]: t=2021-11-16T17:05:43+0000 lvl=warn msg="Failed to get events from member" address=xxx.xx.xx.xx7:8443 err="Unable to connect to: xxx.xx.xx.xx7:8443"
Nov 16 17:05:44 server1 lxd.daemon[2374581]: t=2021-11-16T17:05:44+0000 lvl=warn msg="Failed to get events from member" address=xxx.xx.xx.xx7:8443 err="Unable to connect to: xxx.xx.xx.xx7:8443"
Nov 16 17:05:45 server1 lxd.daemon[2374581]: t=2021-11-16T17:05:45+0000 lvl=warn msg="Failed to get events from member" address=xxx.xx.xx.xx7:8443 err="Unable to connect to: xxx.xx.xx.xx7:8443"
Nov 16 17:05:46 server1 lxd.daemon[2374581]: t=2021-11-16T17:05:46+0000 lvl=warn msg="Failed to get events from member" address=xxx.xx.xx.xx7:8443 err="Unable to connect to: xxx.xx.xx.xx7:8443"
Nov 16 17:05:47 server1 lxd.daemon[2374581]: t=2021-11-16T17:05:47+0000 lvl=warn msg="Failed to get events from member" address=xxx.xx.xx.xx7:8443 err="Unable to connect to: xxx.xx.xx.xx7:8443"
Nov 16 17:08:13 server1 lxd.daemon[2374581]: 2021/11/16 17:08:13 http: TLS handshake error from 192.241.209.114:50642: tls: client offered only unsupported versions: [301]
Nov 16 17:28:36 server1 lxd.daemon[2374581]: 2021/11/16 17:28:36 http: TLS handshake error from 192.241.199.52:45964: tls: client offered only unsupported versions: []
Nov 16 21:23:13 server1 lxd.daemon[2374581]: t=2021-11-16T21:23:13+0000 lvl=warn msg="Failed to get events from member" address=xxx.xx.xx.xx9:8443 err="Unable to connect to: xxx.xx.xx.xx9:8443"
Nov 16 21:23:14 server1 lxd.daemon[2374581]: t=2021-11-16T21:23:14+0000 lvl=warn msg="Failed to get events from member" address=xxx.xx.xx.xx9:8443 err="Unable to connect to: xxx.xx.xx.xx9:8443"
Nov 16 21:23:31 server1 lxd.daemon[2374581]: t=2021-11-16T21:23:31+0000 lvl=warn msg="Excluding offline member from refresh" ID=1 address=xxx.xx.xx.xx9:8443 lastHeartbeat=2021-11-16T21:22:55+0000 raftID=1
Nov 16 21:31:22 server1 lxd.daemon[2374581]: 2021/11/16 21:31:22 http: TLS handshake error from xxx.xx.xx.xx8:59938: EOF
Nov 16 21:31:53 server1 lxd.daemon[2374581]: 2021/11/16 21:31:53 http: TLS handshake error from xxx.xx.xx.xx8:60018: EOF
Nov 16 21:32:05 server1 lxd.daemon[2374581]: 2021/11/16 21:32:05 http: TLS handshake error from xxx.xx.xx.xx8:60040: EOF
Nov 16 21:32:08 server1 lxd.daemon[2374581]: 2021/11/16 21:32:08 http: TLS handshake error from xxx.xx.xx.xx8:60050: EOF
Nov 16 21:32:09 server1 lxd.daemon[2374581]: 2021/11/16 21:32:09 http: TLS handshake error from xxx.xx.xx.xx8:60052: EOF
Nov 16 21:32:10 server1 lxd.daemon[2374581]: 2021/11/16 21:32:10 http: TLS handshake error from xxx.xx.xx.xx8:60060: EOF
Nov 16 21:32:12 server1 lxd.daemon[2374581]: t=2021-11-16T21:32:12+0000 lvl=warn msg="Excluding offline member from refresh" ID=2 address=xxx.xx.xx.xx7:8443 lastHeartbeat=2021-11-16T21:31:45+0000 raftID=2
Nov 16 21:32:12 server1 lxd.daemon[2374581]: t=2021-11-16T21:32:12+0000 lvl=warn msg="Excluding offline member from refresh" ID=1 address=xxx.xx.xx.xx9:8443 lastHeartbeat=2021-11-16T21:31:49+0000 raftID=1
Nov 16 21:32:37 server1 lxd.daemon[2374581]: t=2021-11-16T21:32:37+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:32:12+0000 raftID=4
Nov 16 21:32:39 server1 lxd.daemon[2374581]: t=2021-11-16T21:32:39+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:32:12+0000 raftID=4
Nov 16 21:32:44 server1 lxd.daemon[2374581]: t=2021-11-16T21:32:44+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:32:12+0000 raftID=4
Nov 16 21:32:51 server1 lxd.daemon[2374581]: 2021/11/16 21:32:51 http: TLS handshake error from xxx.xx.xx.xx8:60140: EOF
Nov 16 21:32:56 server1 lxd.daemon[2374581]: t=2021-11-16T21:32:56+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 21:33:29 server1 lxd.daemon[2374581]: t=2021-11-16T21:33:29+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:32:54+0000 raftID=4
Nov 16 21:33:36 server1 lxd.daemon[2374581]: t=2021-11-16T21:33:36+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:32:54+0000 raftID=4
Nov 16 21:33:45 server1 lxd.daemon[2374581]: t=2021-11-16T21:33:45+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:32:54+0000 raftID=4
Nov 16 21:34:14 server1 lxd.daemon[2374581]: 2021/11/16 21:34:14 http: TLS handshake error from xxx.xx.xx.xx8:60268: EOF
Nov 16 21:34:30 server1 lxd.daemon[2374581]: 2021/11/16 21:34:30 http: TLS handshake error from xxx.xx.xx.xx8:60296: EOF
Nov 16 21:37:29 server1 lxd.daemon[2374581]: t=2021-11-16T21:37:29+0000 lvl=eror msg="Failed getting volumes for auto custom volume snapshot task" err="failed to begin transaction: context deadline exceeded"
Nov 16 21:38:22 server1 lxd.daemon[2374581]: 2021/11/16 21:38:22 http: TLS handshake error from xxx.xx.xx.xx8:60656: EOF
Nov 16 21:40:11 server1 lxd.daemon[2374581]: 2021/11/16 21:40:11 http: TLS handshake error from xxx.xx.xx.xx8:60812: EOF
Nov 16 21:41:18 server1 lxd.daemon[2374581]: 2021/11/16 21:41:18 http: TLS handshake error from xxx.xx.xx.xx8:60922: EOF
Nov 16 21:41:29 server1 lxd.daemon[2374581]: 2021/11/16 21:41:29 http: TLS handshake error from xxx.xx.xx.xx8:60946: EOF
Nov 16 21:42:33 server1 lxd.daemon[2374581]: 2021/11/16 21:42:33 http: TLS handshake error from xxx.xx.xx.xx8:32802: EOF
Nov 16 21:43:32 server1 lxd.daemon[2374581]: 2021/11/16 21:43:32 http: TLS handshake error from xxx.xx.xx.xx8:32886: EOF
Nov 16 21:44:00 server1 lxd.daemon[2374581]: t=2021-11-16T21:44:00+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:43:31+0000 raftID=4
Nov 16 21:44:11 server1 lxd.daemon[2374581]: t=2021-11-16T21:44:11+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:43:31+0000 raftID=4
Nov 16 21:44:22 server1 lxd.daemon[2374581]: t=2021-11-16T21:44:22+0000 lvl=warn msg="Failed to rollback transaction after error (Failed to fetch field Devices: Failed to fetch ref for instance_snapshots: sql: transaction has already been committed or rolled back): sql: transaction has already been committed or rolled back"
Nov 16 21:44:22 server1 lxd.daemon[2374581]: t=2021-11-16T21:44:22+0000 lvl=eror msg="Failed to list instance snapshots" err="Failed to fetch field Devices: Failed to fetch ref for instance_snapshots: sql: transaction has already been committed or rolled back" instance=kientran-server1 project=default
Nov 16 21:44:28 server1 lxd.daemon[2374581]: t=2021-11-16T21:44:28+0000 lvl=warn msg="Dqlite: attempt 1: server xxx.xx.xx.xx6:8443: no known leader"
Nov 16 21:44:29 server1 lxd.daemon[2374581]: t=2021-11-16T21:44:29+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 21:44:29 server1 lxd.daemon[2374581]: t=2021-11-16T21:44:29+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 21:44:29 server1 lxd.daemon[2374581]: t=2021-11-16T21:44:29+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 21:44:29 server1 lxd.daemon[2374581]: t=2021-11-16T21:44:29+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 21:44:30 server1 lxd.daemon[2374581]: t=2021-11-16T21:44:30+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 21:44:32 server1 lxd.daemon[2374581]: t=2021-11-16T21:44:32+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 21:44:32 server1 lxd.daemon[2374581]: 2021/11/16 21:44:32 http: TLS handshake error from xxx.xx.xx.xx8:32980: EOF
Nov 16 21:44:59 server1 lxd.daemon[2374581]: 2021/11/16 21:44:59 http: TLS handshake error from xxx.xx.xx.xx8:33022: EOF
Nov 16 21:46:42 server1 lxd.daemon[2374581]: 2021/11/16 21:46:42 http: TLS handshake error from xxx.xx.xx.xx8:33184: EOF
Nov 16 21:47:21 server1 lxd.daemon[2374581]: 2021/11/16 21:47:21 http: TLS handshake error from xxx.xx.xx.xx8:33240: EOF
Nov 16 21:48:33 server1 lxd.daemon[2374581]: 2021/11/16 21:48:33 http: TLS handshake error from xxx.xx.xx.xx8:33338: EOF
Nov 16 21:49:08 server1 lxd.daemon[2374581]: 2021/11/16 21:49:08 http: TLS handshake error from xxx.xx.xx.xx8:33380: EOF
Nov 16 21:49:17 server1 lxd.daemon[2374581]: t=2021-11-16T21:49:17+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:48:50+0000 raftID=4
Nov 16 21:49:32 server1 lxd.daemon[2374581]: t=2021-11-16T21:49:32+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 21:49:34 server1 lxd.daemon[2374581]: t=2021-11-16T21:49:34+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 21:49:34 server1 lxd.daemon[2374581]: t=2021-11-16T21:49:34+0000 lvl=warn msg="Dqlite: attempt 1: server xxx.xx.xx.xx6:8443: no known leader"
Nov 16 21:49:41 server1 lxd.daemon[2374581]: t=2021-11-16T21:49:41+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 21:49:42 server1 lxd.daemon[2374581]: 2021/11/16 21:49:42 http: TLS handshake error from xxx.xx.xx.xx8:33438: EOF
Nov 16 21:49:43 server1 lxd.daemon[2374581]: t=2021-11-16T21:49:43+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 21:49:59 server1 lxd.daemon[2374581]: t=2021-11-16T21:49:59+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 21:51:19 server1 lxd.daemon[2374581]: t=2021-11-16T21:51:19+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:50:51+0000 raftID=4
Nov 16 21:51:27 server1 lxd.daemon[2374581]: 2021/11/16 21:51:27 http: TLS handshake error from xxx.xx.xx.xx8:33608: EOF
Nov 16 21:51:31 server1 lxd.daemon[2374581]: t=2021-11-16T21:51:31+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 21:51:42 server1 lxd.daemon[2374581]: 2021/11/16 21:51:42 http: TLS handshake error from xxx.xx.xx.xx8:33636: EOF
Nov 16 21:52:42 server1 lxd.daemon[2374581]: t=2021-11-16T21:52:42+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:52:10+0000 raftID=4
Nov 16 21:52:44 server1 lxd.daemon[2374581]: 2021/11/16 21:52:44 http: TLS handshake error from xxx.xx.xx.xx8:33726: EOF
Nov 16 21:53:23 server1 lxd.daemon[2374581]: 2021/11/16 21:53:23 http: TLS handshake error from xxx.xx.xx.xx8:33782: EOF
Nov 16 21:54:28 server1 lxd.daemon[2374581]: t=2021-11-16T21:54:28+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:53:59+0000 raftID=4
Nov 16 21:54:40 server1 lxd.daemon[2374581]: 2021/11/16 21:54:40 http: TLS handshake error from xxx.xx.xx.xx8:33894: EOF
Nov 16 21:54:41 server1 lxd.daemon[2374581]: t=2021-11-16T21:54:41+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 21:55:22 server1 lxd.daemon[2374581]: 2021/11/16 21:55:22 http: TLS handshake error from xxx.xx.xx.xx8:33952: EOF
Nov 16 21:55:32 server1 lxd.daemon[2374581]: 2021/11/16 21:55:32 http: TLS handshake error from xxx.xx.xx.xx8:33966: EOF
Nov 16 21:56:24 server1 lxd.daemon[2374581]: 2021/11/16 21:56:24 http: TLS handshake error from 162.142.125.128:8370: read tcp xxx.xx.xx.xx6:8443->162.142.125.128:8370: read: connection reset by peer
Nov 16 21:56:40 server1 lxd.daemon[2374581]: t=2021-11-16T21:56:40+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:56:06+0000 raftID=4
Nov 16 21:57:29 server1 lxd.daemon[2374581]: t=2021-11-16T21:57:29+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:56:58+0000 raftID=4
Nov 16 21:57:40 server1 lxd.daemon[2374581]: t=2021-11-16T21:57:40+0000 lvl=eror msg="Error refreshing forkdns" err="failed to begin transaction: context deadline exceeded"
Nov 16 21:57:53 server1 lxd.daemon[2374581]: 2021/11/16 21:57:53 http: TLS handshake error from xxx.xx.xx.xx8:34202: EOF
Nov 16 21:58:50 server1 lxd.daemon[2374581]: t=2021-11-16T21:58:50+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:58:17+0000 raftID=4
Nov 16 21:59:00 server1 lxd.daemon[2374581]: t=2021-11-16T21:59:00+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:58:17+0000 raftID=4
Nov 16 21:59:42 server1 lxd.daemon[2374581]: t=2021-11-16T21:59:42+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T21:59:10+0000 raftID=4
Nov 16 22:00:00 server1 lxd.daemon[2374581]: 2021/11/16 22:00:00 http: TLS handshake error from xxx.xx.xx.xx8:34402: EOF
Nov 16 22:00:45 server1 lxd.daemon[2374581]: t=2021-11-16T22:00:45+0000 lvl=warn msg="Excluding offline member from refresh" ID=4 address=xxx.xx.xx.xx6:8443 lastHeartbeat=2021-11-16T22:00:19+0000 raftID=4
Nov 16 22:00:50 server1 lxd.daemon[2374581]: t=2021-11-16T22:00:50+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 22:01:01 server1 lxd.daemon[2374581]: t=2021-11-16T22:01:01+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 22:01:01 server1 lxd.daemon[2374581]: t=2021-11-16T22:01:01+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 22:01:03 server1 lxd.daemon[2374581]: t=2021-11-16T22:01:03+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" member=4
Nov 16 22:01:12 server1 systemd[1]: Stopping Service for snap application lxd.daemon...
Nov 16 22:01:13 server1 lxd.daemon[2394805]: => Stop reason is: snap refresh
Nov 16 22:01:13 server1 lxd.daemon[2394805]: => Stopping LXD
Nov 16 22:01:21 server1 lxd.daemon[2374581]: t=2021-11-16T22:01:21+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:01:27 server1 lxd.daemon[2374581]: t=2021-11-16T22:01:27+0000 lvl=eror msg="Failed to start expired instance snapshots operation" err="LXD is shutting down"
Nov 16 22:01:31 server1 lxd.daemon[2374581]: t=2021-11-16T22:01:31+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:01:39 server1 lxd.daemon[2374581]: t=2021-11-16T22:01:39+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:01:46 server1 lxd.daemon[2374581]: t=2021-11-16T22:01:46+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:01:55 server1 lxd.daemon[2374581]: t=2021-11-16T22:01:55+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:02:10 server1 lxd.daemon[2374581]: t=2021-11-16T22:02:10+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:02:19 server1 lxd.daemon[2374581]: t=2021-11-16T22:02:19+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:02:30 server1 lxd.daemon[2374581]: t=2021-11-16T22:02:30+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:02:38 server1 lxd.daemon[2374581]: t=2021-11-16T22:02:38+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:02:50 server1 lxd.daemon[2374581]: t=2021-11-16T22:02:50+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:02:57 server1 lxd.daemon[2374581]: t=2021-11-16T22:02:57+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:03:06 server1 lxd.daemon[2374581]: t=2021-11-16T22:03:06+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:03:17 server1 lxd.daemon[2374581]: t=2021-11-16T22:03:17+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:03:27 server1 lxd.daemon[2374581]: t=2021-11-16T22:03:27+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:03:38 server1 lxd.daemon[2374581]: t=2021-11-16T22:03:38+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:03:46 server1 lxd.daemon[2374581]: t=2021-11-16T22:03:46+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:04:00 server1 lxd.daemon[2374581]: t=2021-11-16T22:04:00+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:04:07 server1 lxd.daemon[2374581]: t=2021-11-16T22:04:07+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:04:21 server1 lxd.daemon[2374581]: t=2021-11-16T22:04:21+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:04:28 server1 lxd.daemon[2374581]: t=2021-11-16T22:04:28+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:04:34 server1 lxd.daemon[2374581]: t=2021-11-16T22:04:34+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:04:47 server1 lxd.daemon[2374581]: t=2021-11-16T22:04:47+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:04:58 server1 lxd.daemon[2374581]: t=2021-11-16T22:04:58+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:05:06 server1 lxd.daemon[2374581]: t=2021-11-16T22:05:06+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:05:17 server1 lxd.daemon[2374581]: t=2021-11-16T22:05:17+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:05:27 server1 lxd.daemon[2374581]: t=2021-11-16T22:05:27+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:05:36 server1 lxd.daemon[2374581]: t=2021-11-16T22:05:36+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:05:50 server1 lxd.daemon[2374581]: t=2021-11-16T22:05:50+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:05:56 server1 lxd.daemon[2374581]: t=2021-11-16T22:05:56+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:06:09 server1 lxd.daemon[2374581]: t=2021-11-16T22:06:09+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:06:17 server1 lxd.daemon[2374581]: t=2021-11-16T22:06:17+0000 lvl=warn msg="Rejecting heartbeat request as shutting down"
Nov 16 22:06:33 server1 lxd.daemon[2394805]: ==> Forcefully stopping LXD after 5 minutes wait
Nov 16 22:06:33 server1 lxd.daemon[2394805]: ==> Stopped LXD
Nov 16 22:06:33 server1 systemd[1]: snap.lxd.daemon.service: Succeeded.
Nov 16 22:06:33 server1 systemd[1]: Stopped Service for snap application lxd.daemon.