Base information
- Distribution: Ubuntu
- Distribution version: 20.04.6 LTS (Focal Fossa)
- Kernel version: Linux etesync 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
- LXC version: 4.0.9
- LXD version: 4.0.9
- Snap revision: 24061
- Storage backend in use: dir
I’ve upgraded the packages in the host (including snap), updated the packages in the container, then rebooted both. Since then the container failed to start.
Upon inspection of the log I was surprised to notice, that the container failed to start because it tried to load the zfs backend (I’ve never used zfs on this host; it is not even installed). : Failed to start the daemon" err="Failed initializing storage pool \"default\": Failed to run: zpool import -f -d /var/snap/lxd/common/lxd/disks default: cannot import 'default': no such pool available"
Then I realized that the directory /var/snap/lxd/common/lxd/storage-pools/default/containers/radicale is empty; there’s no rootfs subfolder (and it is nowhere to be seen). I also performed a search for the “radicale” files throughout the hast’s filesystem, and could not identify any of rootfs’ content. I’ve also compared the size of all the files I could search through with the hard drive size and reported free space, and unfortunately I do not see the expected 3.8GB of unaccounted for data.
I’ve run the extundelete --restore-directory /var/snap/lxd/common/lxd/storage-pools/default/containers/radicale/rootfs /dev/sda to check if it was deleted, but got no trace of it.
The host is running on a Linode (Akamai).
After pulling the backup I can confirm, that the container was never on ZFS.
I have tried to manually correct the backing storage by running these commands as root:
$ sudo sqlite3 /var/snap/lxd/common/lxd/database/global/db.bin "SELECT * FROM storage_pools"
1|default|zfs||1
$ sudo sqlite3 /var/snap/lxd/common/lxd/database/global/db.bin "SELECT * FROM storage_pools_config"
2|1|1|size|20GB
3|1|1|source|/var/snap/lxd/common/lxd/disks/default.img
4|1|1|zfs.pool_name|default
$ sudo sqlite3 /var/snap/lxd/common/lxd/database/global/db.bin "UPDATE storage_pools SET driver='dir' WHERE id=1"
$ sudo sqlite3 /var/snap/lxd/common/lxd/database/global/db.bin "DELETE FROM storage_pools_config WHERE key='size'"
$ sudo sqlite3 /var/snap/lxd/common/lxd/database/global/db.bin "DELETE FROM storage_pools_config WHERE key='zfs.pool_name'"
$ sudo sqlite3 /var/snap/lxd/common/lxd/database/global/db.bin "UPDATE storage_pools_config SET value='/var/snap/lxd/common/lxd/storage-pools/default' WHERE key='source'"
$ sudo sqlite3 /var/snap/lxd/common/lxd/database/global/db.bin "UPDATE storage_pools_config SET id=1 WHERE storage_pool_id=1"
$ sudo sqlite3 /var/snap/lxd/common/lxd/database/global/db.bin "SELECT * FROM storage_pools"
1|default|dir||1
$ sudo sqlite3 /var/snap/lxd/common/lxd/database/global/db.bin "SELECT * FROM storage_pools_config"
1|1|1|source|/var/snap/lxd/common/lxd/storage-pools/default
$ sudo lxd --debug --group lxd
After the operation, the storage reverted itself back to the zfs automatically on its own. Maybe it can be a clue.
Why did the LXD get confused with the storage type?
Is there anything I can do to get the container back?