Help with failed and botched lxd.migrate attempt (LXD 3.0.3 deb to LXD 4.0.9 snap)

How can I best recover from the mess that I’ve made?

I’ve attempted to migrate from lxd 3.0.3 on Ubuntu 18.04 to the snap package. First, lxd.migrate had problems. Then I had problems with the lxc commands not working in the new version. So I uninstalled lxd apt/deb, unaware that lxd.migrate had not completed. That did not help, so I uninstalled he snap version of lxd and re-installed the apt/deb version.

Bottom line is that I think I’ve pretty much lost any chance of getting things back other than a manual recovery of some sort.

At this point, I have the apt/dev package installed. I’m looking for the easiest/fastest way to get my containers back up and running. I’m open to moving forward with either apt/deb or snap packages, whichever is easier & faster. I will eventually want to move to snap, but getting these containers back quickly is important.

My issues with currently installed apt/deb package are:

  • I need help figuring out how to setup and/or recover my storage pools. The current state is that I don’t have any storage pools defined b/c every time I try to create a storage pool and map it to the existing zfs pool (pogo1 fwiw) I get a message that says that the pool is not empty.

  • lxd.migrate removed the mountpoints on my zfs container file systems and I don’t remember and I can’t figure out what those mount points need to look like. The lxd import command says that “you must first mount your container storage at the expected path inside the storage-pools directory”. However, without an example, I don’t know what the “expected path” is.

  • Related to the point above - I’m pretty sure that the snap version brought significant changes in the directory structure of containers etc. If the lxd directory structure changed from version 2 to version 3, it MAY be relevant that my containers started with an early version 2. I say this b/c I don’t remember “storage-pools” being in the mount path on the containers AND it’s not in the storage path of containers that I have on another system that is still at lxd version 2.0.11 on Ubuntu 16.04. The mount paths on that system are “/var/lib/lxd/containers/container_name.zfs”. I believe that this is how my mount paths were on the system I’m working on now.

  • I would like to eventually move my containers to a new zfs filesystem on a zfs pool with newer and hopefully less-likely-to-fail hard drives. Any hints for helping with this would be appreciated.

I believe the files system containers in zfs are untouched other than having their mount points removed. Although I have 20 some containers, I really only care about three of them, and I can recreate the networking configuration if necessary. (They have fixed IP addresses configured inside the containers)

I’m happy to provide additional diagnostic info as needed, though since I’ve uninstalled and reinstalled lxd apt/dev & snap versions several times, I’m guessing it will be useless.

TIA - Jonathan

Here’s the listing of my zfs pool that holds the containers:

root@pogo:~# zfs list |grep pogo1
pogo1                                                                                   492G  1.28T    23K  /pogo1
pogo1/backup                                                                             19K  1.28T    19K  /pogo1/backup
pogo1/containers                                                                        167G  1.28T    19K  /pogo1/containers
pogo1/containers/compiler                                                              1.47G  1.28T  2.00G  none
pogo1/containers/dc3                                                                   2.83G  1.28T  1.84G  none
pogo1/containers/dolibar                                                               2.07G  1.28T  2.25G  /var/lib/lxd/containers/storage-pools/default/dolibar.zfs
pogo1/containers/en2                                                                    282M  1.28T   846M  none
pogo1/containers/erp1                                                                   214M  1.28T   786M  none
pogo1/containers/erp5                                                                  2.53G  1.28T  2.94G  none
pogo1/containers/erp5-1804                                                             6.15G  1.28T  6.57G  none
pogo1/containers/erp5-2                                                                 837M  1.28T  1.23G  none
pogo1/containers/erp5-3                                                                20.0G  1.28T  20.3G  none
pogo1/containers/erp5-4                                                                 841M  1.28T  1.04G  none
pogo1/containers/erp5-buster                                                           1.01G  1.28T  1.30G  none
pogo1/containers/erp5-stretch                                                          1.04G  1.28T  1.29G  none
pogo1/containers/erpnext                                                               4.40G  1.28T  4.84G  none
pogo1/containers/frappe                                                                46.4M  1.28T  1.04G  none
pogo1/containers/fs1                                                                    951M  1.28T  1.12G  none
pogo1/containers/fs2                                                                   43.0G  1.28T  43.6G  none
pogo1/containers/fs3                                                                   42.0G  1.28T  30.6G  none
pogo1/containers/mf                                                                    8.40G  1.28T  8.87G  none
pogo1/containers/ofbiz                                                                 1.57G  1.28T  2.04G  none
pogo1/containers/pervsql                                                               7.98G  1.28T  8.57G  none
pogo1/containers/samba                                                                 14.0G  1.28T  8.99G  none
pogo1/containers/samba-build                                                           1.45G  1.28T  1.62G  none
pogo1/containers/samba-member                                                           388M  1.28T   788M  none
pogo1/containers/samba465                                                              1.26G  1.28T  1.74G  none
pogo1/containers/weberp                                                                1.56G  1.28T  1.77G  none
pogo1/containers/wins                                                                   539M  1.28T   884M  none
pogo1/custom                                                                             24K  1.28T    24K  none
pogo1/deleted                                                                          10.2G  1.28T    19K  /pogo1/deleted
pogo1/deleted/containers                                                                 24K  1.28T    24K  none
pogo1/deleted/custom                                                                     24K  1.28T    24K  none
pogo1/deleted/images                                                                   10.2G  1.28T    19K  /pogo1/deleted/images
pogo1/deleted/images/2cd33749a175551616b2024877a83e5b5b405ffe4b1e76427ad92726536d429c   266M  1.28T   266M  none
pogo1/deleted/images/5b72cf46f628b3d60f5d99af48633539b2916993c80fc5a2323d7d841f66afbe   666M  1.28T   666M  none
pogo1/deleted/images/7a4f7e85f1faf3ee349cbbb80422ba229f27339a749ca2ba8dc099d463329bc2   641M  1.28T   641M  none
pogo1/deleted/images/7a7ff654cbd8f5f09bec03aa19d8d7d92649127d18659036a963b1ea63f90d25   585M  1.28T   585M  none
pogo1/deleted/images/7d72e477a2f43fa37a649675158e41921cb20d5199ef57204ce36442e3f50df1   484M  1.28T   484M  none
pogo1/deleted/images/87a1f1c305615024124b238c0fcbed99c11193cab4bbc5c340ca2500c99b1b1f   686M  1.28T   686M  none
pogo1/deleted/images/96e12fc44b24f052b5f959137fabff715b83856a8a5eb64fbc1338d3f173a82e   688M  1.28T   688M  none
pogo1/deleted/images/98dc4bd7cca2fcd24c82ee403baaca3a5bd1845f925540786d0954a5973c181c   643M  1.28T   643M  none
pogo1/deleted/images/b190d5ec0c537468465e7bd122fe127d9f3509e3a09fb699ac33b0c5d4fe050f   635M  1.28T   635M  none
pogo1/deleted/images/b46db39f29fa786c854e226975f21c1d98a218eed73471876d9f079a42e2781d   795M  1.28T   795M  none
pogo1/deleted/images/b8b83fee4399a09be187488de56f27bed673592dde6b3db74094498cf7bef1e0   929M  1.28T   929M  none
pogo1/deleted/images/be7cec7c948958adfbb9bc7dbd292762d2388cc883466815fc2b6bc06bf06f5a   599M  1.28T   599M  none
pogo1/deleted/images/c395a7105278712478ec1dbfaab1865593fc11292f99afe01d5b94f1c34a9a3a   640M  1.28T   640M  none
pogo1/deleted/images/d59c1830aabe183897507784d0da4480af73e9d0bd2c83c87eadacfce7d0164b   308M  1.28T   308M  none
pogo1/deleted/images/d6f281a2e523674bcd9822f3f61be337c51828fb0dc94c8a200ab216d12a0fff   659M  1.28T   659M  none
pogo1/deleted/images/ee5129ec7855ae748a64d001a0f5de0f21740752175ce2223873da9d53fc3af1   608M  1.28T   608M  none
pogo1/deleted/images/f2228450779fee27020d6024af587379b8f51062c32a335327f2b028c924bfa1   605M  1.28T   605M  none
pogo1/deleted/virtual-machines                                                           24K  1.28T    24K  none
pogo1/gateway2                                                                         32.9G  1.28T  32.9G  /pogo1/gateway2
pogo1/gateway3                                                                         75.7G  1.28T  75.7G  /pogo1/gateway3
pogo1/home                                                                               19K  1.28T    19K  /pogo1/home
pogo1/images                                                                           2.20G  1.28T    19K  /pogo1/images
pogo1/images/2bf21e674c6da5faeaae6554316fca60fcae99e3bd002a9e9db4e1782a2ba85c          1.00G  1.28T  1.00G  none
pogo1/images/51d8adf6ba25c3f79963994a06cbfb55aa6eb2ebb2d67817bd0938b799ec0315           639M  1.28T   639M  none
pogo1/images/8220e89e33e6f62b56cb451cfed61574074416a66a6e7c61ff574d95572e6661           586M  1.28T   586M  none
pogo1/kfse_backups                                                                       19K  1.28T    19K  /pogo1/kfse_backups
pogo1/kms_backup                                                                         19K  1.28T    19K  /pogo1/kms_backup
pogo1/oldpooh2                                                                          182G  1.28T   175G  /pogo1/oldpooh2
pogo1/psql                                                                             13.9G  1.28T  7.52G  /pogo1/psql
pogo1/psql-attempt1                                                                    7.52G  1.28T  7.51G  /pogo1/psql-attempt1
pogo1/snapshots                                                                          98K  1.28T    26K  /pogo1/snapshots
pogo1/snapshots/dc3                                                                      24K  1.28T    24K  /pogo1/snapshots/dc3
pogo1/snapshots/fs3                                                                      24K  1.28T    24K  /pogo1/snapshots/fs3
pogo1/snapshots/samba                                                                    24K  1.28T    24K  /pogo1/snapshots/samba
pogo1/virtual-machines                                                                   24K  1.28T    24K  none
root@pogo:~#

The mountpoint on the dolibar container is my attempt to make something work. lxd.migrate removed the mountpoint and I created the one above to attemp a recovery.

What problems? Using lxd.migrate was the correct approach, so would be useful to understand what went wrong here.

I was smart enough to collect this info:

root@pogo:~# lxd.migrate
=> Connecting to source server
=> Connecting to destination server
=> Running sanity checks

=== Source server
LXD version: 3.0.3
LXD PID: 5099
Resources:
  Containers: 26
  Images: 2
  Networks: 0
  Storage pools: 2

=== Destination server
LXD version: 4.0.9
LXD PID: 28045
Resources:
  Containers: 0
  Images: 0
  Networks: 0
  Storage pools: 0

The migration process will shut down all your containers then move your data to the destination LXD.
Once the data is moved, the destination LXD will start and apply any needed updates.
And finally your containers will be brought back to their previous state, completing the migration.

Are you ready to proceed (yes/no) [default=no]? y
=> Shutting down the source LXD
=> Stopping the source LXD units
=> Stopping the destination LXD unit
=> Unmounting source LXD paths
=> Unmounting destination LXD paths
=> Wiping destination LXD clean
=> Backing up the database
=> Moving the data
=> Updating the storage backends
error: Failed to update the storage pools: Failed to run: nsenter --mount=/run/snapd/ns/lxd.mnt zfs set mountpoint=/var/lib/snapd/hostfs/pogo1 pogo1: umount: /var/lib/snapd/hostfs/pogo1: target is busy.
cannot unmount '/var/lib/snapd/hostfs/pogo1': umount failed
cannot mount '/pogo1/backup': failed to create mountpoint
cannot mount '/pogo1/containers': failed to create mountpoint
cannot mount '/pogo1/deleted': failed to create mountpoint
cannot mount '/pogo1/deleted/images': failed to create mountpoint
cannot mount '/pogo1/gateway2': failed to create mountpoint
cannot mount '/pogo1/gateway3': failed to create mountpoint
cannot mount '/pogo1/home': failed to create mountpoint
cannot mount '/pogo1/images': failed to create mountpoint
cannot mount '/pogo1/kfse_backups': failed to create mountpoint
cannot mount '/pogo1/kms_backup': failed to create mountpoint
cannot mount '/pogo1/oldpooh2': failed to create mountpoint
cannot mount '/pogo1/psql-attempt1': failed to create mountpoint
cannot mount '/pogo1/snapshots': failed to create mountpoint
cannot mount '/pogo1/snapshots/dc3': failed to create mountpoint
cannot mount '/pogo1/snapshots/fs3': failed to create mountpoint
cannot mount '/pogo1/snapshots/samba': failed to create mountpoint
property may be set but unable to remount filesystem

Keep in mind that I’ve uninstalled both apt/deb and snap versions AFTER I ran lxd.migrate. (not the smartest move in retrospect) So I doubt we’ll be able to recover anything from them.

OK so looks like at least part of the issue was related to:

Thats what the Failed to update the storage pools error was from. Although not sure why it hung then.

What I suggest you do now is to sudo snap install lxd and then run sudo lxd recover.

This will ask you for the source of the pool, which you would enter as pogo1 and when it asks for additional pool settings enter: zfs.pool_name=pogo1 as well.

This may be able to recover the instances from the backup.yaml files on disk.

Oh and please remove any custom mount points and set them all to mountpoint=legacy first.

Do I need to remove the empty lxd 3.0.3 apt/deb first before install the snap version? Also, which snap version do you recommend that I install?

Yes remove the deb for now (dont run it with --purge though).

Suggest using the version you installed previously.

I initially installed/removed snap version 4.09. But before I realized I needed to ask for help, I also installed/remove the default version which I believe is 5.6. Does it matter? If not, I just a soon have the latest version unless one is better at the recovery process that we are attempting.

Would suggest LXD 5.6 then, sudo snap install lxd and then use sudo lxd.recover to see if it can detect anything useful.

Also, here is a listing of the pogo1 pool. Note that I had created other non-lxd related file systems on this pool. I’ve not changed their mountpoints. I’ve set all the mountpoints that lxd.migrate set to “none” back to “legacy”. They were not set to “legacy” originally with lxd, but I’m taking your word that his is the best way to recover the containers.

root@pogo:/etc/network# zfs list|grep pogo1
pogo1                                                                                   492G  1.28T    23K  /pogo1
pogo1/backup                                                                             19K  1.28T    19K  /pogo1/backup
pogo1/containers                                                                        167G  1.28T    19K  /pogo1/containers
pogo1/containers/compiler                                                              1.47G  1.28T  2.00G  legacy
pogo1/containers/dc3                                                                   2.83G  1.28T  1.84G  legacy
pogo1/containers/dolibar                                                               2.07G  1.28T  2.25G  legacy
pogo1/containers/en2                                                                    282M  1.28T   846M  legacy
pogo1/containers/erp1                                                                   214M  1.28T   786M  legacy
pogo1/containers/erp5                                                                  2.53G  1.28T  2.94G  legacy
pogo1/containers/erp5-1804                                                             6.15G  1.28T  6.57G  legacy
pogo1/containers/erp5-2                                                                 837M  1.28T  1.23G  legacy
pogo1/containers/erp5-3                                                                20.0G  1.28T  20.3G  legacy
pogo1/containers/erp5-4                                                                 841M  1.28T  1.04G  legacy
pogo1/containers/erp5-buster                                                           1.01G  1.28T  1.30G  legacy
pogo1/containers/erp5-stretch                                                          1.04G  1.28T  1.29G  legacy
pogo1/containers/erpnext                                                               4.40G  1.28T  4.84G  legacy
pogo1/containers/frappe                                                                46.4M  1.28T  1.04G  legacy
pogo1/containers/fs1                                                                    951M  1.28T  1.12G  legacy
pogo1/containers/fs2                                                                   43.0G  1.28T  43.6G  legacy
pogo1/containers/fs3                                                                   42.0G  1.28T  30.6G  legacy
pogo1/containers/mf                                                                    8.40G  1.28T  8.87G  legacy
pogo1/containers/ofbiz                                                                 1.57G  1.28T  2.04G  legacy
pogo1/containers/pervsql                                                               7.98G  1.28T  8.57G  legacy
pogo1/containers/samba                                                                 14.0G  1.28T  8.99G  legacy
pogo1/containers/samba-build                                                           1.45G  1.28T  1.62G  legacy
pogo1/containers/samba-member                                                           388M  1.28T   788M  legacy
pogo1/containers/samba465                                                              1.26G  1.28T  1.74G  legacy
pogo1/containers/weberp                                                                1.56G  1.28T  1.77G  legacy
pogo1/containers/wins                                                                   539M  1.28T   884M  legacy
pogo1/custom                                                                             24K  1.28T    24K  legacy
pogo1/deleted                                                                          10.2G  1.28T    19K  /pogo1/deleted
pogo1/deleted/containers                                                                 24K  1.28T    24K  legacy
pogo1/deleted/custom                                                                     24K  1.28T    24K  legacy
pogo1/deleted/images                                                                   10.2G  1.28T    19K  /pogo1/deleted/images
pogo1/deleted/images/2cd33749a175551616b2024877a83e5b5b405ffe4b1e76427ad92726536d429c   266M  1.28T   266M  legacy
pogo1/deleted/images/5b72cf46f628b3d60f5d99af48633539b2916993c80fc5a2323d7d841f66afbe   666M  1.28T   666M  legacy
pogo1/deleted/images/7a4f7e85f1faf3ee349cbbb80422ba229f27339a749ca2ba8dc099d463329bc2   641M  1.28T   641M  legacy
pogo1/deleted/images/7a7ff654cbd8f5f09bec03aa19d8d7d92649127d18659036a963b1ea63f90d25   585M  1.28T   585M  legacy
pogo1/deleted/images/7d72e477a2f43fa37a649675158e41921cb20d5199ef57204ce36442e3f50df1   484M  1.28T   484M  legacy
pogo1/deleted/images/87a1f1c305615024124b238c0fcbed99c11193cab4bbc5c340ca2500c99b1b1f   686M  1.28T   686M  legacy
pogo1/deleted/images/96e12fc44b24f052b5f959137fabff715b83856a8a5eb64fbc1338d3f173a82e   688M  1.28T   688M  legacy
pogo1/deleted/images/98dc4bd7cca2fcd24c82ee403baaca3a5bd1845f925540786d0954a5973c181c   643M  1.28T   643M  legacy
pogo1/deleted/images/b190d5ec0c537468465e7bd122fe127d9f3509e3a09fb699ac33b0c5d4fe050f   635M  1.28T   635M  legacy
pogo1/deleted/images/b46db39f29fa786c854e226975f21c1d98a218eed73471876d9f079a42e2781d   795M  1.28T   795M  legacy
pogo1/deleted/images/b8b83fee4399a09be187488de56f27bed673592dde6b3db74094498cf7bef1e0   929M  1.28T   929M  legacy
pogo1/deleted/images/be7cec7c948958adfbb9bc7dbd292762d2388cc883466815fc2b6bc06bf06f5a   599M  1.28T   599M  legacy
pogo1/deleted/images/c395a7105278712478ec1dbfaab1865593fc11292f99afe01d5b94f1c34a9a3a   640M  1.28T   640M  legacy
pogo1/deleted/images/d59c1830aabe183897507784d0da4480af73e9d0bd2c83c87eadacfce7d0164b   308M  1.28T   308M  legacy
pogo1/deleted/images/d6f281a2e523674bcd9822f3f61be337c51828fb0dc94c8a200ab216d12a0fff   659M  1.28T   659M  legacy
pogo1/deleted/images/ee5129ec7855ae748a64d001a0f5de0f21740752175ce2223873da9d53fc3af1   608M  1.28T   608M  legacy
pogo1/deleted/images/f2228450779fee27020d6024af587379b8f51062c32a335327f2b028c924bfa1   605M  1.28T   605M  legacy
pogo1/deleted/virtual-machines                                                           24K  1.28T    24K  legacy
pogo1/gateway2                                                                         32.9G  1.28T  32.9G  /pogo1/gateway2
pogo1/gateway3                                                                         75.7G  1.28T  75.7G  /pogo1/gateway3
pogo1/home                                                                               19K  1.28T    19K  /pogo1/home
pogo1/images                                                                           2.20G  1.28T    19K  /pogo1/images
pogo1/images/2bf21e674c6da5faeaae6554316fca60fcae99e3bd002a9e9db4e1782a2ba85c          1.00G  1.28T  1.00G  legacy
pogo1/images/51d8adf6ba25c3f79963994a06cbfb55aa6eb2ebb2d67817bd0938b799ec0315           639M  1.28T   639M  legacy
pogo1/images/8220e89e33e6f62b56cb451cfed61574074416a66a6e7c61ff574d95572e6661           586M  1.28T   586M  legacy
pogo1/kfse_backups                                                                       19K  1.28T    19K  /pogo1/kfse_backups
pogo1/kms_backup                                                                         19K  1.28T    19K  /pogo1/kms_backup
pogo1/oldpooh2                                                                          182G  1.28T   175G  /pogo1/oldpooh2
pogo1/psql                                                                             13.9G  1.28T  7.52G  /pogo1/psql
pogo1/psql-attempt1                                                                    7.52G  1.28T  7.51G  /pogo1/psql-attempt1
pogo1/snapshots                                                                          98K  1.28T    26K  /pogo1/snapshots
pogo1/snapshots/dc3                                                                      24K  1.28T    24K  /pogo1/snapshots/dc3
pogo1/snapshots/fs3                                                                      24K  1.28T    24K  /pogo1/snapshots/fs3
pogo1/snapshots/samba                                                                    24K  1.28T    24K  /pogo1/snapshots/samba
pogo1/virtual-machines                                                                   24K  1.28T    24K  legacy
root@pogo:/etc/network#
1 Like

I removed apt/deb lxd

root@pogo:~# apt remove lxd lxd-client
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  ebtables libuv1 xdelta3
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  lxd lxd-client
0 upgraded, 0 newly installed, 2 to remove and 4 not upgraded.
After this operation, 31.7 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 119948 files and directories currently installed.)
Removing lxd (3.0.3-0ubuntu1~18.04.2) ...
Removing lxd-client (3.0.3-0ubuntu1~18.04.2) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1.6) ...
You have new mail in /var/mail/root
root@pogo:~#

Then installed snap lxd

root@pogo:~# snap install lxd
lxd 5.6-794016a from Canonical✓ installed
You have new mail in /var/mail/root
root@pogo:~#

Then tried to run lxd.recover and got this:

root@pogo:~# lxd.recover
lxd.recover: command not found
root@pogo:~#

So I logged out of my console and logged back in to make sure everything is fresh. I still get the same result. So I checked the version of LXD to make sure it’s there:

root@pogo:~# lxd version
5.6
root@pogo:~#

And checked lxc to make sure it’s there:

root@pogo:~# lxc list
+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+
root@pogo:~#

and the version…

root@pogo:~# lxc --version
5.6
root@pogo:~#

What next?

Apologies try sudo lxd recover.

@Tomp Oops. The command should be lxd recover, not lxd.recover. You wrote it correctly the first time, but I saw the one with the dot in it.

I obviously don’t know how to enter the correct information that lxd recover is asking for. I tried to find docs online for it with no success.

root@pogo:~# lxd recover
This LXD server currently has the following storage pools:
Would you like to recover another storage pool? (yes/no) [default=no]: y
Name of the storage pool: pogo1
Name of the storage backend (btrfs, ceph, cephfs, cephobject, dir, lvm, zfs): zfs
Source of the storage pool (block device, volume group, dataset, path, ... as applicable): ^C
You have new mail in /var/mail/root
root@pogo:~# lxd recover
This LXD server currently has the following storage pools:
Would you like to recover another storage pool? (yes/no) [default=no]: y
Name of the storage pool: default
Name of the storage backend (btrfs, ceph, cephfs, cephobject, dir, lvm, zfs): zfs
Source of the storage pool (block device, volume group, dataset, path, ... as applicable): pogo1
Additional storage pool configuration property (KEY=VALUE, empty when done):
Would you like to recover another storage pool? (yes/no) [default=no]:
The recovery process will be scanning the following storage pools:
 - NEW: "default" (backend="zfs", source="pogo1")
Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]: y
Scanning for unknown volumes...
Error: Failed validation request: Failed mounting pool "default": Cannot mount pool as "zfs.pool_name" is not specified
root@pogo:~#

What is the “name of the storage pool”? The LXD pool? The ZFS pool?
What is the “source of the storage pool” in my case? “dataset”?
Do I need to specify any KEY=value combinations? I’m sure I used the default settings when I originally setup lxd 2.0

I would be happy to experiment and try something until it works - except that’s the behavior that got me into this mess.

BTW, Thanks a bundle for your help!

When it asks:

Additional storage pool configuration property (KEY=VALUE, empty when done):

Enter:

zfs.pool_name=pogo1

See also LXD recover doesn't use storage volume defaults logic · Issue #10862 · lxc/lxd · GitHub for background on why this is needed.

So I tried again and got a little bit further, but it is still failing.

root@pogo:~# lxd recover
This LXD server currently has the following storage pools:
Would you like to recover another storage pool? (yes/no) [default=no]: y
Name of the storage pool: default
Name of the storage backend (btrfs, ceph, cephfs, cephobject, dir, lvm, zfs): zfs
Source of the storage pool (block device, volume group, dataset, path, ... as applicable): pogo1
Additional storage pool configuration property (KEY=VALUE, empty when done): zfs.pool_name=pogo1
Additional storage pool configuration property (KEY=VALUE, empty when done):
Would you like to recover another storage pool? (yes/no) [default=no]: n
The recovery process will be scanning the following storage pools:
 - NEW: "default" (backend="zfs", source="pogo1")
Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]: y
Scanning for unknown volumes...
Error: Failed validation request: Failed mounting pool "default": Failed to run: zfs set acltype=posixacl xattr=sa atime=off relatime=on mountpoint=legacy setuid=on exec=on devices=on pogo1: exit status 1 (umount: /var/lib/snapd/hostfs/pogo1: target is busy.
cannot unmount '/var/lib/snapd/hostfs/pogo1': umount failed
cannot mount '/pogo1/backup': failed to create mountpoint
cannot mount '/pogo1/containers': failed to create mountpoint
cannot mount '/pogo1/deleted': failed to create mountpoint
cannot mount '/pogo1/deleted/images': failed to create mountpoint
cannot mount '/pogo1/gateway2': failed to create mountpoint
cannot mount '/pogo1/gateway3': failed to create mountpoint
cannot mount '/pogo1/home': failed to create mountpoint
cannot mount '/pogo1/images': failed to create mountpoint
cannot mount '/pogo1/kfse_backups': failed to create mountpoint
cannot mount '/pogo1/kms_backup': failed to create mountpoint
cannot mount '/pogo1/oldpooh2': failed to create mountpoint
cannot mount '/pogo1/psql-attempt1': failed to create mountpoint
cannot mount '/pogo1/snapshots': failed to create mountpoint
cannot mount '/pogo1/snapshots/dc3': failed to create mountpoint
cannot mount '/pogo1/snapshots/fs3': failed to create mountpoint
cannot mount '/pogo1/snapshots/samba': failed to create mountpoint
property may be set but unable to remount filesystem)
You have new mail in /var/mail/root
root@pogo:~#

pogo1 does not exist in /var/lib/snapd/hostfs/

root@pogo:~# cd /var/lib/snapd/hostfs/
root@pogo:/var/lib/snapd/hostfs# ll
total 8
drwxr-xr-x  2 root root 4096 Oct  1 21:39 ./
drwxr-xr-x 23 root root 4096 Oct  3 11:23 ../
root@pogo:/var/lib/snapd/hostfs#

I can also confirm that the mount points are still set to “legacy”.

Now what?

So I finally realized that I needed to remove all custom point mounts on my zfs file system and set them to “legacy” to get the lxd recover command to work. Until then, I kept getting umount errors from lxd recover.

However, I’m still getting errors and lxd recover is not finding my containers.

root@pogo:~# lxd recover
This LXD server currently has the following storage pools:
Would you like to recover another storage pool? (yes/no) [default=no]: y
Name of the storage pool: default
Name of the storage backend (btrfs, ceph, cephfs, cephobject, dir, lvm, zfs): zfs
Source of the storage pool (block device, volume group, dataset, path, ... as applicable): pogo1
Additional storage pool configuration property (KEY=VALUE, empty when done): zfs.pool_name=pogo1
Additional storage pool configuration property (KEY=VALUE, empty when done):
Would you like to recover another storage pool? (yes/no) [default=no]: n
The recovery process will be scanning the following storage pools:
 - NEW: "default" (backend="zfs", source="pogo1")
Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]: y
Scanning for unknown volumes...
Error: Failed validation request: Failed checking volumes on pool "default": Failed parsing backup file "/var/snap/lxd/common/lxd/storage-pools/default/containers/erp1/backup.yaml": open /var/snap/lxd/common/lxd/storage-pools/default/containers/erp1/backup.yaml: no such file or directory
root@pogo:~#

So what are:

  • Failed validation request? I presume this is internal b/c nothing was presented to the console.
  • Failed checking volumes on pool “default”? Is this the lxd pool or the zfs pool?
  • Failed parsing backup file “/var/snap/lxd/common/lxd/storage-pools/default/containers/erp1/backup.yaml”? Of course this would fail as it does not exist.

Am I not entering the correct information in the lxd recover session?

All of the filesystems on pogo1 have their mountpoint set to legacy and are unmounted. I’m assuming that lxd recover handles mounting and unmounting the zfs filesystems to find the info that it’s looking for. However, all the filesystems remain unmounted after the command executes, so either the command unmounts the filesystems or they were never successfully mounted.

To test for the existance of the backup.yaml file in a container, I tried to mount it using mount, but it failed with “canonicalization error 2”. After trying to research this and finding no working examples, I gave up and set a zfs mountpoint and zfs mounted the filesystem. I can confirm that the backup.yaml file exists in the container filesystem, so lxd recover is not yet able to find it.

What am I doing wrong here?

In case it helps, here’s my system info:
OS = Ubuntu 18.04.4 LTS
LXD version = 5.6
ZFS version info below:

root@pogo:~# dmesg | grep ZFS
[  246.835181] ZFS: Loaded module v0.7.5-1ubuntu16.12, ZFS pool version 5000, ZFS filesystem version 5

Please could you copy one of the backup.yaml files here for inspection?

Also do you recall what the LXD name of your pogo1 ZFS pool was?

Was it “default” or “pogo1”?
If its the latter then try entering Name of the storage pool: pogo1 instead.

Also does /var/snap/lxd/common/lxd/storage-pools/default or /var/snap/lxd/common/lxd/storage-pools/pogo1 exist?

The error is showing the hierarchy of what is failing:

It is failing to validate the recover request (as lxd recover does a non-destructive scan of the storage devices before recovering anything into the DB), the specific thing that is failing is that its failing to check volumes for pool “default” (which is what you said was the LXD name of the pool in the form). Then the reason its failing to check the volumes is that its failing to open and parse the first container’s backup.yaml file because for some reason it doesn’t exist or it cant find it.