LXD recovery after HDD failure

A few days ago I was able to backup my SSD short before failure. The system was already in a quite unstable state with services like ssh failing. I was not able to export the containers properly anymore. LXD reported some integrity issues and crashed. Not sure how faulty is the disk dump, but I hope for the best regarding the zfs pool.

I would like to recover the containers in a different machine. So far I was able to:

  • mount the root partition incl. var/snap/lxd/common/lxd/database
  • mount the zfs pool as read-only under the temporary name backup (since default is already taken on this machine), so I also have access to all files in the containers
  • lxd from snap was already present at the rescue machine

I looked at the following answer to a similar question, but I am lost in the last part:

  1. snap remove lxd
  2. snap install lxd <-- is this correct?
  3. zfs destroy -r default/lxd
  4. zfs rename backup/lxd default/lxd
  5. snap install lxd
  6. nsenter + mkdir + mount the datasets <-- I’m lost here!
  7. lxd import container1, etc. for all containers

So I can follow the first half, but I think renaming the pool should happen before installing lxd again. I am quite lost in what to do in 6:

  • what is the nsenter good for?
  • to what directory should I mount the containers?
  • is it possible to take advantage of the lxd database in the backup or any other container related data during the restore?

I would be very happy about any help, since I’ve never done this before and I don’t want do destroy any of the rescued data by accident. Thank you in advance.

For anyone who encounters the same situation, here is my successful solution. I needed a few days to find out the right steps and their ordering, fighting various error messages.

Recovery containers from a damaged drive, assuming you managed to backup/dump the zfs partition on a second HDD that is now in a new machine with Ubuntu 20.04, goes as follows:

  1. sudo snap install lxd && sudo apt install zfsutils-linux
    installs the necessary stuff
  2. sudo adduser $(whoami) lxd && newgrp lxd
    adds you to the lxd group granting you permissions to manage LXD
  3. sudo zpool import
    prints a list of available volumes to import, remember a long number at the top, e.g. 1234567890123456789
  4. sudo zpool import -d /dev -f -R / 1234567890123456789 default
    imports the volume, note that long number copied from the listing in the previous step
    – if you just want to read the files, you can also add -o readonly=on but that will not allow you to import any container
    – i chose -R / since the mountpoints for all containers were already correctly specified as /var/snap/lxd/common/lxd/storage-pools/default/containers/container_name (after zpool import just check by zfs list otherwise export and import again with a new path (or just reboot)
    – the last argument is the pool name, use the same as you had on the old machine
  5. Maybe you need to call lxc info once, not sure but it will do no harm anyway.
  6. create any profiles you used in the old setup, e.g. X11
  7. sudo zfs mount default/containers/container_name
  8. sudo lxd import --force container_name
  9. sudo zfs unmount default/containers/container_name
  10. repeat 7. - 9. for all containers you need to recover

lxc list should now show all recovered containers, note that after starting their IP address will not be assigned. Now you may either export the containers, re-install lxd (now finally calling lxd init as well) and import the images again or you may follow the suggestions over here to initialize network and storage manually:

Fingers crossed for your recovery!

For step 7 you can check the mounting with
zfs list -ro canmount,mounted,mountpoint,name