Hi
I’m trying to recover a number of containers from a zfs pool I’ve imported into a new 22.04 installation. I’ve read a few threads here but there is quite a bit of variance and none exactly match my case. Moreover, I can’t seem to apply any of the techniques mentioned here with any success. Even though many of my error messages are the same as reported in other postings. When using lxd recover I really don’t know what I am supposed to enter at some of the prompts and so am really guessing at what it is asking for. Not disimilar to other postings I’ve read. Consequently I would be grateful for any help or advise including pointers to useful doc’s. I’m finding it hard to find anything but titbits on this subject. And unfortunately these span several different versions of LXD, LXC, SNAP and Ubuntu generally.
History:
I installed Ubuntu 20.04 onto a usb stick and used it to boot an old server. I had two hard drives in the machine which I used to create a zfs mirrored pool. I created a number of containers on that pool for learning and testing purposes. Then my USB stick developed an error and I did not have a recent backup.
Recovery attempts.
I created a new 20.04 usb stick and booted the machine. I imported the zpool without any issues but couldn’t get lxd to import or recover my containers.
I read a number of posts and the subject of bugs and patches came up on situations similar to mine. Consequently I thought I might as well upgrade to 22.04 and fix the problem there.
I created a new 22.04 USB stick and tried again. But the same thing happened. I imported my zpool without any issues but experienced the same issues I read about when it came to importing/recovering containers eg;
LXD init not loading my pool as it had data on it.
LXD recover not finding my pool at all.
LXD recover giving me various errors when I try to ‘directed’ it to my pool.
I’ve tried mounting some of my containers in various places but get similar errors reported by others.
I am not supprised it’s not working as I very much feel like I am hacking away blindly in the dark. In the absense of substantive documentation It’s very much a case of ‘monkey read, monkey type’, with the addition of some ill informed guess work. If there are any “comprehensive” docs on the subject of lxd recovery I’d very much appreciate a link.
Technical Details:-
Versions:
cat /etc/lsb-release
--------------------
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
root@dmz-001:~# zfs --version
-------------------------------
zfs-2.1.4-0ubuntu0.1
zfs-kmod-2.1.4-0ubuntu0.1
root@dmz-001:~# lxd --version
-----------------------------
5.0.2
root@dmz-001:~# snap --version
--------------------------------------------------
snap 2.58+22.04
snapd 2.58+22.04
series 16
ubuntu 22.04
kernel 5.15.0-58-generic
Zpool Details
-------------------------
root@dmz-001:~# zpool list
-------------------------------------------------------------------------------------------------------
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
dmz-pool-001 464G 37.0G 427G 39.6M - 4% 7% 1.00x ONLINE -
------------------------------
root@dmz-001:~# zpool status
-------------------------------------------------------------------------------------------------------
pool: dmz-pool-001
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:11:16 with 0 errors on Sun Jan 8 04:18:51 2023
checkpoint: created Thu Feb 2 18:02:28 2023, consumes 39.6M
config:
NAME STATE READ WRITE CKSUM
dmz-pool-001 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-Hitachi_HDS721050CLA362_JP1570HR1KEGRK ONLINE 0 0 0
ata-Hitachi_HDS721050CLA362_JP1570HR1MJM4K ONLINE 0 0 0
errors: No known data errors
ZFS Data Sets
--------------------------
root@dmz-001:~# zfs list
-------------------------------------------------------------------------------------------------------
NAME USED AVAIL REFER MOUNTPOINT
dmz-pool-001 70.9G 379G 57K /dmz-pool-001
dmz-pool-001/lxd-backups 2.44G 379G 2.44G /dmz-pool-001/lxd-backups
dmz-pool-001/lxd-default 19.5G 379G 25K /dmz-pool-001/lxd-default
dmz-pool-001/lxd-default/containers 9.85G 379G 24K /dmz-pool-001/lxd-default/containers
dmz-pool-001/lxd-default/containers/Drupal-1 95.3M 379G 370M none
dmz-pool-001/lxd-default/containers/FDNS 422M 379G 406M none
dmz-pool-001/lxd-default/containers/NTP 171M 379G 286M
/var/snap/lxd/common/lxd/storage-pools/default/containers/A-NTP
dmz-pool-001/lxd-default/containers/RDNS 182M 379G 276M none
dmz-pool-001/lxd-default/containers/ADNS 247M 379G 448M none
dmz-pool-001/lxd-default/containers/Drupal-2 142M 379G 325M none
dmz-pool-001/lxd-default/containers/Drupal-3 265M 379G 2.44G /mnt/Drupal-3
dmz-pool-001/lxd-default/containers/PGA 487M 379G 802M none
dmz-pool-001/lxd-default/containers/ISDB 189M 379G 404M /mnt/ISDB
dmz-pool-001/lxd-default/containers/Rsyslog 2.59M 379G 262M none
dmz-pool-001/lxd-default/containers/Postgres 1.91G 379G 1.24G none
dmz-pool-001/lxd-default/containers/Mail 621M 379G 566M none
dmz-pool-001/lxd-default/deleted 4.80G 379G 24K none
dmz-pool-001/lxd-default/deleted/containers 3.90G 379G 24K none
dmz-pool-001/lxd-default/deleted/containers/8114a5d7-bcaf-4098-92ee-1761bba6b5ba 393M 379G 263M none
dmz-pool-001/lxd-default/deleted/containers/bf29f6fa-def8-4961-995c-7e45f587c253 42.4M 379G 203M none
dmz-pool-001/lxd-default/deleted/custom 24K 379G 24K none
dmz-pool-001/lxd-default/deleted/images 927M 379G 24K none
dmz-pool-001/lxd-default/deleted/virtual-machines 24K 379G 24K none
dmz-pool-001/lxd-default/deleted/containers/fa49f8ca-d01e-4e8d-b036-b6f1d152fda3 3.47G 379G 4.00G
/var/snap/lxd/common/lxd/storage-pools/default/containers/ubuntu-testing
dmz-pool-001/lxd-default/deleted/images/144df099adfdd60d49939ef1cddbeb8029a09729ef5521224fc91b7f19625ef5 457M 379G 457M
/var/snap/lxd/common/lxd/storage-pools/default/images/144df099adfdd60d49939ef1cddbeb8029a09729ef5521224fc91b7f19625ef5
dmz-pool-001/lxd-default/deleted/images/475d2b7886dcd2b252e26c5f30674bb7212780da2a2d273ff285f4fff0c0ff41 163M 379G 163M none
dmz-pool-001/lxd-default/deleted/images/c53a85d3e035eb0bc6f3e54c92fedcbe823b112026631d0c11df9211e43d3215 307M 379G 307M
/var/snap/lxd/common/lxd/storage-pools/default/images/c53a85d3e035eb0bc6f3e54c92fedcbe823b112026631d0c11df9211e43d3215
dmz-pool-001/lxd-default/images 4.88G 379G 24K /dmz-pool-001/lxd-default/images
dmz-pool-001/lxd-default/images/4efbe3f8531565b1885086e70338cba8345fa8869fef8c5ea2eaea281fb2e331 2.44G 379G 2.44G
/var/snap/lxd/common/lxd/storage-pools/default/images/4efbe3f8531565b1885086e70338cba8345fa8869fef8c5ea2eaea281fb2e331
dmz-pool-001/lxd-default/images/9ddb41abb98d326a915ff6a061888f863c18a7817070189dbd997bb8f5644c54 2.44G 379G 2.44G
/var/snap/lxd/common/lxd/storage-pools/default/images/9ddb41abb98d326a915ff6a061888f863c18a7817070189dbd997bb8f5644c54
dmz-pool-001/lxd-default/virtual-machines 24K 379G 24K /dmz-pool-001/lxd-default/virtual-machines
dmz-pool-001/lxd-live-images 24K 379G 24K /dmz-pool-001/lxd-live-images
dmz-pool-001/lxd-template-images 24K 379G 24K /dmz-pool-001/lxd-template-images
dmz-pool-001/lxd-testarea 6.43G 379G 6.43G /dmz-pool-001/lxd-testarea
dmz-pool-001/swap 34.0G 413G 12K -
dmz-pool-001/usb-stick 8.49G 379G 8.49G /dmz-pool-001/usb-stick-data
ZFS Mounted Datasets:
--------------------------------------
root@dmz-001:~# zfs mount
-------------------------------------------------------------------------------------------------------------
dmz-pool-001 /dmz-pool-001
dmz-pool-001/lxd-live-images /dmz-pool-001/lxd-live-images
dmz-pool-001/lxd-testarea /dmz-pool-001/lxd-testarea
dmz-pool-001/lxd-backups /dmz-pool-001/lxd-backups
dmz-pool-001/lxd-default /dmz-pool-001/lxd-default
dmz-pool-001/lxd-template-images /dmz-pool-001/lxd-template-images
dmz-pool-001/usb-stick-data /dmz-pool-001/usb-stick-data
dmz-pool-001/lxd-default/images /dmz-pool-001/lxd-default/images
dmz-pool-001/lxd-default/containers /dmz-pool-001/lxd-default/containers
dmz-pool-001/lxd-default/virtual-machines /dmz-pool-001/lxd-default/virtual-machines
dmz-pool-001/lxd-default/containers/ISDB /mnt/ISDB
Check for actual data at mount points:
--------------------------------------
root@dmz-001:~# ls /mnt/ISDB/
-------------------------------------------------------------------------------------------------------------
backup.yaml metadata.yaml rootfs templates
Attempt at Importing from a mount point:
-------------------------------------------------------------------------------------------------------------
root@dmz-001:~# lxc import /mnt/ISDB/
Error: Post "http://unix.socket/1.0/instances": read /var/lib/snapd/hostfs/mnt/ISDB: is a directory
Attempt at Recovering the pool:
-------------------------------------------------------------------------------------------------------
root@dmz-001:~# lxd recover
This LXD server currently has the following storage pools:
Would you like to recover another storage pool? (yes/no) [default=no]: yes
Name of the storage pool: dmz-pool-001
Name of the storage backend (dir, lvm, zfs, btrfs, ceph, cephfs, cephobject): zfs
Source of the storage pool (block device, volume group, dataset, path, ... as applicable): block device
Additional storage pool configuration property (KEY=VALUE, empty when done):
Would you like to recover another storage pool? (yes/no) [default=no]:
The recovery process will be scanning the following storage pools:
- NEW: "dmz-pool-001" (backend="zfs", source="block device")
Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]:
Scanning for unknown volumes...
Error: Failed validation request: Failed mounting pool "dmz-pool-001": Cannot mount pool as "zfs.pool_name" is not specified
Please note:
I have not yet upgraded ZFS as I don’t know how robust this process is or whether I will need to down grade Ubuntu or one of LXD’s components.
Any help Gratefully received.
Thanks in advance.