Error: Load storage pool for container : No such object

Hello,

After upgrading debian 9 to 10, lxd do not start anymore :frowning:
Here is the error:
EROR[07-19|20:53:49] Failed to start the daemon: Load storage pool for container “vm647606” in project “default”: No such object
Error: Load storage pool for container “vm647606” in project “default”: No such object

This is lxd 3.8 .

Anyone could help ?

Give us some information about the storage pool. Is it ZFS, btrfs, dir or something else?

My wild guess is that the storage pool is using ZFS and somehow your new kernel does not have ZFS support.
It is likely that with the old kernel, you have installed ZFS using, for example, the zfs-dkms package, but on the new kernel this package has not been setup. If that is the case, then configure properly ZFS on the new kernel (maybe sudo apt install zfs-dkms), then restart your computer.

It’s ZFS.

zfs is working with no problem for me. there are many vms on the host, and only this one have error.

# zfs list | grep vm647606
default/containers/vm647606                                                               119M  9.88G   254M  /var/lib/lxd/storage-pools/default/containers/vm647606
default/custom/vm647606                                                                    19K  29.3G    19K  /var/lib/lxd/storage-pools/default/custom/vm647606
# lxd -d -v
INFO[07-20|07:40:32] LXD 3.8 is starting in normal mode       path=/var/lib/lxd
INFO[07-20|07:40:32] Kernel uid/gid map:
INFO[07-20|07:40:32]  - u 0 0 4294967295
INFO[07-20|07:40:32]  - g 0 0 4294967295
INFO[07-20|07:40:32] Configured LXD uid/gid map:
INFO[07-20|07:40:32]  - u 0 1000000 65536
INFO[07-20|07:40:32]  - g 0 1000000 65536
WARN[07-20|07:40:32] AppArmor support has been disabled because of lack of kernel support
INFO[07-20|07:40:32] Kernel features:
INFO[07-20|07:40:32]  - netnsid-based network retrieval: no
INFO[07-20|07:40:32]  - uevent injection: no
INFO[07-20|07:40:32]  - unprivileged file capabilities: no
INFO[07-20|07:40:32] Initializing local database
DBUG[07-20|07:40:32] Initializing database gateway
DBUG[07-20|07:40:32] Start database node                      id=1 address=
DBUG[07-20|07:40:32] Raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:0}]
DBUG[07-20|07:40:32] Raft: Node at 0 [Leader] entering Leader state
DBUG[07-20|07:40:32] Dqlite: starting event loop
DBUG[07-20|07:40:32] Dqlite: accepting connections
INFO[07-20|07:40:32] Starting /dev/lxd handler:
INFO[07-20|07:40:32]  - binding devlxd socket                 socket=/var/lib/lxd/devlxd/sock
INFO[07-20|07:40:32] REST API daemon:
INFO[07-20|07:40:32]  - binding Unix socket                   socket=/var/lib/lxd/unix.socket
INFO[07-20|07:40:32]  - binding TCP socket                    socket=[::]:8448
INFO[07-20|07:40:32] Initializing global database
DBUG[07-20|07:40:32] Dqlite: handling new connection (fd=19)
DBUG[07-20|07:40:32] Dqlite: connected address=0 attempt=0
INFO[07-20|07:40:32] Initializing storage pools
DBUG[07-20|07:40:32] Initializing and checking storage pool "default"
DBUG[07-20|07:40:32] Checking ZFS storage pool "default"
INFO[07-20|07:40:32] Applying patch: storage_api_permissions
DBUG[07-20|07:40:32] Mounting ZFS storage volume "references" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "references" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm613997" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm613997" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm615814" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm615814" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm618252" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm618252" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm619684" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm619684" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm620349" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm620349" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm620362" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm620362" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm621372" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm621372" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm622677" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm622677" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm622846" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm622846" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm623965" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm623965" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm626732" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm626732" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm628679" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm628679" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm628947" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm628947" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm629927" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm629927" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm630487" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm630487" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm631081" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm631081" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm631671" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm631671" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm633999" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm633999" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm635132" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm635132" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm635976" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm635976" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm637440" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm637440" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm637956" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm637956" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm640262" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm640262" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm640342" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm640342" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm640477" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm640477" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm640709" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm640709" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm641369" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm641369" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm641376" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm641376" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm641407" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm641407" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm641983" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm641983" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm642484" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm642484" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm642997" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm642997" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm643333" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm643333" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm643699" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm643699" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm643962" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm643962" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm644034" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm644034" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm644345" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm644345" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm644540" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm644540" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm644856" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm644856" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm644872" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm644872" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm645017" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm645017" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm645134" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm645134" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm645638" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm645638" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm645760" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm645760" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm645936" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm645936" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm646091" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm646091" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm646109" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm646109" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm646260" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm646260" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm646533" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm646533" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm646708" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm646708" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume "vm647546" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume "vm647546" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume for container "vm613997" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume for container "vm613997" on storage pool "default"
DBUG[07-20|07:40:32] Unmounting ZFS storage volume for container "vm613997" on storage pool "default"
DBUG[07-20|07:40:32] Unmounted ZFS storage volume for container "vm613997" on storage pool "default"
DBUG[07-20|07:40:32] Mounting ZFS storage volume for container "vm615814" on storage pool "default"
DBUG[07-20|07:40:32] Mounted ZFS storage volume for container "vm615814" on storage pool "default"
DBUG[07-20|07:40:32] Unmounting ZFS storage volume for container "vm615814" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm615814" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm618252" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm618252" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm618252" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm618252" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm619684" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm619684" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm619684" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm619684" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm620349" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm620349" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm620349" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm620349" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm620362" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm620362" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm620362" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm620362" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm621372" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm621372" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm621372" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm621372" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm622677" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm622677" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm622677" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm622677" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm622846" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm622846" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm622846" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm622846" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm623965" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm623965" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm623965" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm623965" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm626732" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm626732" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm626732" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm626732" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm628679" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm628679" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm628679" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm628679" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm628947" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm628947" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm628947" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm628947" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm629927" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm629927" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm629927" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm629927" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm630487" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm630487" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm630487" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm630487" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm631081" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm631081" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm631081" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm631081" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm631671" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm631671" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm631671" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm631671" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm633999" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm633999" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm633999" on storage pool "default"
DBUG[07-20|07:40:33] Unmounted ZFS storage volume for container "vm633999" on storage pool "default"
DBUG[07-20|07:40:33] Mounting ZFS storage volume for container "vm635132" on storage pool "default"
DBUG[07-20|07:40:33] Mounted ZFS storage volume for container "vm635132" on storage pool "default"
DBUG[07-20|07:40:33] Unmounting ZFS storage volume for container "vm635132" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm635132" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm635976" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm635976" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm635976" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm635976" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm637440" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm637440" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm637440" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm637440" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm637956" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm637956" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm637956" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm637956" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm640262" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm640262" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm640262" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm640262" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm640342" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm640342" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm640342" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm640342" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm640477" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm640477" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm640477" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm640477" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm640709" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm640709" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm640709" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm640709" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm641369" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm641369" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm641369" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm641369" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm641376" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm641376" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm641376" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm641376" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm641407" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm641407" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm641407" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm641407" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm641983" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm641983" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm641983" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm641983" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm642484" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm642484" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm642484" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm642484" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm642997" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm642997" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm642997" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm642997" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm643333" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm643333" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm643333" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm643333" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm643699" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm643699" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm643699" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm643699" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm643962" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm643962" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm643962" on storage pool "default"
DBUG[07-20|07:40:34] Unmounted ZFS storage volume for container "vm643962" on storage pool "default"
DBUG[07-20|07:40:34] Mounting ZFS storage volume for container "vm644034" on storage pool "default"
DBUG[07-20|07:40:34] Mounted ZFS storage volume for container "vm644034" on storage pool "default"
DBUG[07-20|07:40:34] Unmounting ZFS storage volume for container "vm644034" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume for container "vm644034" on storage pool "default"
DBUG[07-20|07:40:35] Mounting ZFS storage volume for container "vm644345" on storage pool "default"
DBUG[07-20|07:40:35] Mounted ZFS storage volume for container "vm644345" on storage pool "default"
DBUG[07-20|07:40:35] Unmounting ZFS storage volume for container "vm644345" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume for container "vm644345" on storage pool "default"
DBUG[07-20|07:40:35] Mounting ZFS storage volume for container "vm644540" on storage pool "default"
DBUG[07-20|07:40:35] Mounted ZFS storage volume for container "vm644540" on storage pool "default"
DBUG[07-20|07:40:35] Unmounting ZFS storage volume for container "vm644540" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume for container "vm644540" on storage pool "default"
DBUG[07-20|07:40:35] Mounting ZFS storage volume for container "vm644856" on storage pool "default"
DBUG[07-20|07:40:35] Mounted ZFS storage volume for container "vm644856" on storage pool "default"
DBUG[07-20|07:40:35] Unmounting ZFS storage volume for container "vm644856" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume for container "vm644856" on storage pool "default"
DBUG[07-20|07:40:35] Mounting ZFS storage volume for container "vm644872" on storage pool "default"
DBUG[07-20|07:40:35] Mounted ZFS storage volume for container "vm644872" on storage pool "default"
DBUG[07-20|07:40:35] Unmounting ZFS storage volume for container "vm644872" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume for container "vm644872" on storage pool "default"
DBUG[07-20|07:40:35] Mounting ZFS storage volume for container "vm645017" on storage pool "default"
DBUG[07-20|07:40:35] Mounted ZFS storage volume for container "vm645017" on storage pool "default"
DBUG[07-20|07:40:35] Unmounting ZFS storage volume for container "vm645017" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume for container "vm645017" on storage pool "default"
DBUG[07-20|07:40:35] Mounting ZFS storage volume for container "vm645134" on storage pool "default"
DBUG[07-20|07:40:35] Mounted ZFS storage volume for container "vm645134" on storage pool "default"
DBUG[07-20|07:40:35] Unmounting ZFS storage volume for container "vm645134" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume for container "vm645134" on storage pool "default"
DBUG[07-20|07:40:35] Mounting ZFS storage volume for container "vm645638" on storage pool "default"
DBUG[07-20|07:40:35] Mounted ZFS storage volume for container "vm645638" on storage pool "default"
DBUG[07-20|07:40:35] Unmounting ZFS storage volume for container "vm645638" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume for container "vm645638" on storage pool "default"
DBUG[07-20|07:40:35] Mounting ZFS storage volume for container "vm645760" on storage pool "default"
DBUG[07-20|07:40:35] Mounted ZFS storage volume for container "vm645760" on storage pool "default"
DBUG[07-20|07:40:35] Unmounting ZFS storage volume for container "vm645760" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume for container "vm645760" on storage pool "default"
DBUG[07-20|07:40:35] Mounting ZFS storage volume for container "vm645936" on storage pool "default"
DBUG[07-20|07:40:35] Mounted ZFS storage volume for container "vm645936" on storage pool "default"
DBUG[07-20|07:40:35] Unmounting ZFS storage volume for container "vm645936" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume for container "vm645936" on storage pool "default"
DBUG[07-20|07:40:35] Mounting ZFS storage volume for container "vm646091" on storage pool "default"
DBUG[07-20|07:40:35] Mounted ZFS storage volume for container "vm646091" on storage pool "default"
DBUG[07-20|07:40:35] Unmounting ZFS storage volume for container "vm646091" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume for container "vm646091" on storage pool "default"
DBUG[07-20|07:40:35] Mounting ZFS storage volume for container "vm646109" on storage pool "default"
DBUG[07-20|07:40:35] Mounted ZFS storage volume for container "vm646109" on storage pool "default"
DBUG[07-20|07:40:35] Unmounting ZFS storage volume for container "vm646109" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume for container "vm646109" on storage pool "default"
DBUG[07-20|07:40:35] Mounting ZFS storage volume for container "vm646260" on storage pool "default"
DBUG[07-20|07:40:35] Mounted ZFS storage volume for container "vm646260" on storage pool "default"
DBUG[07-20|07:40:35] Unmounting ZFS storage volume for container "vm646260" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume for container "vm646260" on storage pool "default"
DBUG[07-20|07:40:35] Mounting ZFS storage volume for container "vm646533" on storage pool "default"
DBUG[07-20|07:40:35] Mounted ZFS storage volume for container "vm646533" on storage pool "default"
DBUG[07-20|07:40:35] Unmounting ZFS storage volume for container "vm646533" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume for container "vm646533" on storage pool "default"
DBUG[07-20|07:40:35] Mounting ZFS storage volume for container "vm646708" on storage pool "default"
DBUG[07-20|07:40:35] Mounted ZFS storage volume for container "vm646708" on storage pool "default"
DBUG[07-20|07:40:35] Unmounting ZFS storage volume for container "vm646708" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume for container "vm646708" on storage pool "default"
DBUG[07-20|07:40:35] Mounting ZFS storage volume for container "vm647546" on storage pool "default"
DBUG[07-20|07:40:35] Mounted ZFS storage volume for container "vm647546" on storage pool "default"
DBUG[07-20|07:40:35] Unmounting ZFS storage volume for container "vm647546" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume for container "vm647546" on storage pool "default"
DBUG[07-20|07:40:35] Database error: &errors.errorString{s:"No such object"}
DBUG[07-20|07:40:35] Unmounting ZFS storage volume "vm647546" on storage pool "default"
DBUG[07-20|07:40:35] Unmounted ZFS storage volume "vm647546" on storage pool "default"
DBUG[07-20|07:40:35] Unmounting ZFS storage volume "vm646708" on storage pool "default"
WARN[07-20|07:40:46] Failed to unmount ZFS filesystem via zfs unmount: umount: /var/lib/lxd/storage-pools/default/custom/vm646708: target is busy.
cannot unmount '/var/lib/lxd/storage-pools/default/custom/vm646708': umount failed
. Trying lazy umount (MNT_DETACH)...
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm646708" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm646533" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm646533" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm646260" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm646260" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm646109" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm646109" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm646091" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm646091" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm645936" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm645936" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm645760" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm645760" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm645638" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm645638" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm645134" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm645134" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm645017" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm645017" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm644856" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm644856" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm644540" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm644540" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm644345" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm644345" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm644034" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm644034" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm643962" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm643962" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm643699" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm643699" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm642997" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm642997" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm642484" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm642484" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm641983" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm641983" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm641407" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm641407" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm641376" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm641376" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm641369" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm641369" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm640477" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm640477" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm640342" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm640342" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm640262" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm640262" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm637956" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm637956" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm635976" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm635976" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm631081" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm631081" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm628947" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm628947" on storage pool "default"
DBUG[07-20|07:40:46] Unmounting ZFS storage volume "vm626732" on storage pool "default"
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm626732" on storage pool "default"
EROR[07-20|07:40:46] Failed to start the daemon: Load storage pool for container "vm647606" in project "default": No such object
INFO[07-20|07:40:46] Starting shutdown sequence
INFO[07-20|07:40:46] Stopping REST API handler:
INFO[07-20|07:40:46]  - closing socket                        socket=[::]:8448
INFO[07-20|07:40:46]  - closing socket                        socket=/var/lib/lxd/unix.socket
INFO[07-20|07:40:46] Stopping /dev/lxd handler:
INFO[07-20|07:40:46]  - closing socket                        socket=/var/lib/lxd/devlxd/sock
INFO[07-20|07:40:46] Closing the database
DBUG[07-20|07:40:46] Dqlite: closing client
DBUG[07-20|07:40:46] Stop database gateway
DBUG[07-20|07:40:46] Stop raft instance
DBUG[07-20|07:40:46] Dqlite: stopping event loop
DBUG[07-20|07:40:46] Dqlite: event loop stopped
INFO[07-20|07:40:46] Unmounting temporary filesystems
INFO[07-20|07:40:46] Done unmounting temporary filesystems
INFO[07-20|07:40:46] Saving simplestreams cache
INFO[07-20|07:40:46] Saved simplestreams cache
Error: Load storage pool for container "vm647606" in project "default": No such object

Okay, the issue that you are facing, is with some problem relating to that single container.

There is the following, which has the same error message with later below.

DBUG[07-20|07:40:35] Database error: &errors.errorString{s:"No such object"}

and the following which resolved itself,

DBUG[07-20|07:40:35] Unmounting ZFS storage volume "vm646708" on storage pool "default"
WARN[07-20|07:40:46] Failed to unmount ZFS filesystem via zfs unmount: umount: /var/lib/lxd/storage-pools/default/custom/vm646708: target is busy.
cannot unmount '/var/lib/lxd/storage-pools/default/custom/vm646708': umount failed
. Trying lazy umount (MNT_DETACH)...
DBUG[07-20|07:40:46] Unmounted ZFS storage volume "vm646708" on storage pool "default"

Your container with the problem is vm647606,

EROR[07-20|07:40:46] Failed to start the daemon: Load storage pool for container "vm647606" in project "default": No such object

and

Error: Load storage pool for container "vm647606" in project "default": No such object

I do not know what could have gone wrong. Perhaps someone else could help here.

ok :disappointed_relieved:
We have many vms blocked :confused:
Any idea to fix ?

Could I just delete this vm in DB so I can start lxd and others vm ?

Can you send me a tarball of /var/lib/lxd/database (if I remember correctly you’re not using the snap) to stgraber at ubuntu dot com

I should be able to force the database to load so I can see if there’s anything odd with that container in the database.

Ok, sent, thank you

Thanks, well received but sadly haven’t had time to load it and poke at it today.

You mentioned that the ability to delete affected containers may be helpful to you.
This could be done (preferably after a backup), by creating /var/lib/lxd/database/patch.global.sql, containing:

DELETE FROM containers WHERE name='name-of-container';
DELETE FROM storage_volumes WHERE name='name-of-container' AND type=0;

This does assume that you don’t have multiple projects with conflicting container names (or you’ll need to tweak the WHERE clause to account for that).

It will delete the container and all related entries from the database, but it will not remove any snapshot for the container, so you may end up with an inconsistent database if you don’t also manually remove those snapshots and matching storage volume for the snapshots.

I’ll make some time to poke at your database tomorrow to see what’s going on with it, hopefully with that, we can take a more targeted approach and just fix whatever needs fixing. There are a number of data migration patches that you need to go through during that startup which all require the database and on-disk state to line up. Most likely you have a database record for something which no longer exists on disk or the reverse case where you have extra bits on disk with no match DB record (would actually line up better with the error).

Ok.

The patch worked perfectly, lxd started, thank you so much !