I noticed that our containers were not purging their snapshots after their expiration.
I thought maybe upgrading the snap to 5.3 “5.3-924be6a” might help, but now all of our containers are in ERROR state.
lxc monitor --type=logging --pretty shows:
WARNING[2022-06-27T22:27:30-07:00] Transaction timed out. Retrying once err="Failed to begin transaction: context deadline exceeded" member=1 DEBUG [2022-06-27T22:27:30-07:00] Database error err="sql: transaction has already been committed or rolled back" WARNING[2022-06-27T22:27:30-07:00] Failed to rollback transaction after error (Failed to fetch from "config" table: sql: Rows are closed): sql: transaction has already been committed or rolled back DEBUG [2022-06-27T22:27:30-07:00] Database error err="Failed to fetch from \"config\" table: sql: Rows are closed" DEBUG [2022-06-27T22:27:30-07:00] Dqlite: attempt 1: server 1: connected DEBUG [2022-06-27T22:27:30-07:00] GetInstanceUsage started instance=container project=default DEBUG [2022-06-27T22:27:30-07:00] GetInstanceUsage finished instance=container project=default WARNING[2022-06-27T22:27:40-07:00] Transaction timed out. Retrying once err="Failed to begin transaction: context deadline exceeded" member=1
Anyway to recover /rebuild database?
I’m fearful to stop other containers since I just tried with a testing one and now it’s unable to start again:
user@srv5:~$ lxc stop u14
user@srv5:~$ lxc start u14
Error: Failed to run: zfs set mountpoint=legacy canmount=noauto data/lxc/containers/u14: umount: /var/snap/lxd/common/shmounts/storage-pools/data/containers/u14: no mount point specified.
cannot unmount ‘/var/snap/lxd/common/shmounts/storage-pools/data/containers/u14’: umount failed
lxc info --show-log u14for more info
1 user@srv5:~$ lxc info --show-log u14
Created: 2022/06/14 17:25 PDT
Last Used: 2022/06/14 17:25 PDT