Post-Snap / 20.04 migration LXD fails to start with patch failure on "storage api rename container snapshots links"

Hi,

I recently upgraded to 20.04. As part of it, I migrated to the 4.0 snap during the release upgrade from 3.0.3 to 4.0.4. The migration hung at the “Waiting for LXD to come online” step. In order to continue with the upgrade, I had to kill the migration pid (after waiting overnight for it to run). The resulting crash was filed as LP #1908460.

Now, on start, we fail with:

t=2020-12-17T02:32:14+0000 lvl=eror msg="Failed to start the daemon: Failed applying patch \"storage_api_rename_container_snapshots_links\": readlink /var/snap/lxd/common/lxd/snapshots/matrix: invalid argument"   
Error: Failed applying patch "storage_api_rename_container_snapshots_links": readlink /var/snap/lxd/common/lxd/snapshots/matrix: invalid argument                                                                    

I have ZFS configured at the storage backend. Here’s the relevant debug info:

lfaraone@lithium:~$ sudo zfs list | grep matrix
tank/lxd/containers/matrix                                                                14.0G   750G  7.37G  /var/snap/lxd/common/lxd/storage-pools/default/containers/hdd-matrix
tank/lxd/snapshots/matrix                                                                  565K   750G   128K  none
tank/lxd/snapshots/matrix/snap0                                                            181K   750G   224K  /var/snap/lxd/common/lxd/snapshots/matrix/snap0
tank/lxd/snapshots/matrix/snap2                                                            256K   750G   362K  /var/snap/lxd/common/lxd/snapshots/matrix/snap2
Dec 17 02:32:12 lithium systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 3.                                                                                                                                                     
Dec 17 02:32:12 lithium systemd[1]: Stopped Service for snap application lxd.daemon.                                                                                                                                                                             
Dec 17 02:32:12 lithium systemd[1]: Started Service for snap application lxd.daemon.                                                                                                                                                                             
Dec 17 02:32:12 lithium lxd.daemon[113755]: => Preparing the system (18150)                                                                                                                                                                                      
Dec 17 02:32:12 lithium lxd.daemon[113755]: ==> Loading snap configuration                                                                                                                                                                                       
Dec 17 02:32:12 lithium lxd.daemon[113755]: ==> Setting up mntns symlink (mnt:[4026532898])                                                                                                                                                                      
Dec 17 02:32:12 lithium lxd.daemon[113755]: ==> Setting up kmod wrapper                                                                                                                                                                                          
Dec 17 02:32:12 lithium lxd.daemon[113755]: ==> Preparing /boot                                                                                                                                                                                                  
Dec 17 02:32:12 lithium lxd.daemon[113755]: ==> Preparing a clean copy of /run                                                                                                                                                                                   
Dec 17 02:32:12 lithium lxd.daemon[113755]: ==> Preparing /run/bin                                                                                                                                                                                               
Dec 17 02:32:12 lithium lxd.daemon[113755]: ==> Preparing a clean copy of /etc                                                                                                                                                                                   
Dec 17 02:32:12 lithium lxd.daemon[113755]: ==> Preparing a clean copy of /usr/share/misc                                                                                                                                                                        
Dec 17 02:32:12 lithium lxd.daemon[113755]: ==> Setting up ceph configuration                                                                                                                                                                                    
Dec 17 02:32:12 lithium lxd.daemon[113755]: ==> Setting up LVM configuration                                                                                                                                                                                     
Dec 17 02:32:12 lithium lxd.daemon[113755]: ==> Rotating logs                                                                                                                                                                                                    
Dec 17 02:32:12 lithium lxd.daemon[113755]: ==> Setting up ZFS (0.7)                                                                                                                                                                                             
Dec 17 02:32:12 lithium lxd.daemon[113755]: ==> Escaping the systemd cgroups                                                                                                                                                                                     
Dec 17 02:32:12 lithium lxd.daemon[113755]: ====> Detected cgroup V1                                                                                                                                                                                             
Dec 17 02:32:12 lithium lxd.daemon[113755]: ==> Escaping the systemd process resource limits                                                                                                                                                                     
Dec 17 02:32:12 lithium lxd.daemon[113755]: ==> Disabling shiftfs on this kernel (auto)                                                                                                                                                                          
Dec 17 02:32:12 lithium lxd.daemon[5647]: Closed liblxcfs.so                                                                                                                                                                                                     
Dec 17 02:32:12 lithium lxd.daemon[5647]: Running destructor lxcfs_exit                                                                                                                                                                                          
Dec 17 02:32:12 lithium lxd.daemon[5647]: Running constructor lxcfs_init to reload liblxcfs                                                                                                                                                                      
Dec 17 02:32:12 lithium kernel: new mount options do not match the existing superblock, will be ignored                                                                                                                                                          
Dec 17 02:32:12 lithium lxd.daemon[5647]: mount namespace: 5                                                                                                                                                                                                     
Dec 17 02:32:12 lithium lxd.daemon[5647]: hierarchies:                                                                                                                                                                                                           
Dec 17 02:32:12 lithium lxd.daemon[5647]:   0: fd:   6:                                                                                                                                                                                                          
Dec 17 02:32:12 lithium lxd.daemon[5647]:   1: fd:   7: name=systemd                                                                                                                                                                                             
Dec 17 02:32:12 lithium lxd.daemon[5647]:   2: fd:   8: pids                                                                                                                                                                                                     
Dec 17 02:32:12 lithium lxd.daemon[5647]:   3: fd:   9: devices                                                                                                                                                                                                  
Dec 17 02:32:12 lithium lxd.daemon[5647]:   4: fd:  10: rdma                                                                                                                                                                                                     
Dec 17 02:32:12 lithium lxd.daemon[5647]:   5: fd:  11: freezer                                                                                                                                                                                                  
Dec 17 02:32:12 lithium lxd.daemon[5647]:   6: fd:  12: hugetlb                                                                                                                                                                                                  
Dec 17 02:32:12 lithium lxd.daemon[5647]:   7: fd:  13: cpu,cpuacct                                                                                                                                                                                              
Dec 17 02:32:12 lithium lxd.daemon[5647]:   8: fd:  14: memory                                                                                                                                                                                                   
Dec 17 02:32:12 lithium lxd.daemon[5647]:   9: fd:  15: perf_event                                                                                                                                                                                               
Dec 17 02:32:12 lithium lxd.daemon[5647]:  10: fd:  16: cpuset                                                                                                                                                                                                   
Dec 17 02:32:12 lithium lxd.daemon[5647]:  11: fd:  17: net_cls,net_prio                                                                                                                                                                                         
Dec 17 02:32:12 lithium lxd.daemon[5647]:  12: fd:  19: blkio                                                                                                                                                                                                    
Dec 17 02:32:12 lithium lxd.daemon[5647]: Kernel does not support swap accounting                                                                                                                                                                                
Dec 17 02:32:12 lithium lxd.daemon[5647]: api_extensions:                                                                                                                                                                                                        
Dec 17 02:32:12 lithium lxd.daemon[5647]: - cgroups                                                                                                                                                                                                              
Dec 17 02:32:12 lithium lxd.daemon[5647]: - sys_cpu_online                                                                                                                                                                                                       
Dec 17 02:32:12 lithium lxd.daemon[5647]: - proc_cpuinfo                                                                                                                                                                                                         
Dec 17 02:32:12 lithium lxd.daemon[5647]: - proc_diskstats                                                                                                                                                                                                       
Dec 17 02:32:12 lithium lxd.daemon[5647]: - proc_loadavg                                                                                                                                                                                                         
Dec 17 02:32:12 lithium lxd.daemon[5647]: - proc_meminfo                                                                                                                                                                                                         
Dec 17 02:32:12 lithium lxd.daemon[5647]: - proc_stat                                                                                                                                                                                                            
Dec 17 02:32:12 lithium lxd.daemon[5647]: - proc_swaps                                                                                                                                                                                                           
Dec 17 02:32:12 lithium lxd.daemon[5647]: - proc_uptime                                                                                                                                                                                                          
Dec 17 02:32:12 lithium lxd.daemon[5647]: - shared_pidns                                                                                                                                                                                                         
Dec 17 02:32:12 lithium lxd.daemon[5647]: - cpuview_daemon                                                                                                                                                                                                       
Dec 17 02:32:12 lithium lxd.daemon[5647]: - loadavg_daemon                                                                                                                                                                                                       
Dec 17 02:32:12 lithium lxd.daemon[5647]: - pidfds                                                                                                                                                                                                               
Dec 17 02:32:12 lithium lxd.daemon[5647]: Reloaded LXCFS                                                                                                                                                                                                         
Dec 17 02:32:12 lithium lxd.daemon[113755]: => Re-using existing LXCFS                                                                                                                                                                                           
Dec 17 02:32:12 lithium lxd.daemon[113755]: => Starting LXD                                                                                                                                                                                                      
Dec 17 02:32:13 lithium lxd.daemon[113872]: t=2020-12-17T02:32:13+0000 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"                                                                                            
Dec 17 02:32:13 lithium lxd.daemon[113872]: t=2020-12-17T02:32:13+0000 lvl=eror msg="Failed to start the daemon: Failed applying patch \"storage_api_rename_container_snapshots_links\": readlink /var/snap/lxd/common/lxd/snapshots/matrix: invalid argument"   
Dec 17 02:32:13 lithium lxd.daemon[113872]: Error: Failed applying patch "storage_api_rename_container_snapshots_links": readlink /var/snap/lxd/common/lxd/snapshots/matrix: invalid argument                                                                    
Dec 17 02:32:13 lithium lxd.daemon[113755]: => LXD failed to start                                                                                                                                                                                               
Dec 17 02:32:13 lithium systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE                                                                                                                                                  
Dec 17 02:32:13 lithium systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.                                                                                                                                                                     
Dec 17 02:32:14 lithium systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 4.                                                                                                                                                     
Dec 17 02:32:14 lithium systemd[1]: Stopped Service for snap application lxd.daemon.                                                                                                                                                                             

Hmm, not sure what’s going on, /var/snap/lxd/common/lxd/snapshots even on 3.0 should only ever contain symlinks to the storage pool, it sounds like this isn’t the case on your system.

It’s also weird that you have anything under lxd/snapshots/* in your pool.
Can you show zfs list -t all?

I think we’ll first want to cleanup the ZFS state to bring things back to normal, then go cleanup /var/snap/lxd/common/lxd/snapshots to just contain symlinks to the relevant entries under /var/snap/lxd/common/lxd/storage-pools/, that should make the upgrade logic work again.

1 Like

thanks stgraber for taking a look! :slight_smile:

Hmm, not sure what’s going on, /var/snap/lxd/common/lxd/snapshots even on 3.0 should only ever contain symlinks to the storage pool, it sounds like this isn’t the case on your system.

Yeah, that’s accurate:

lfaraone@lithium:~$ sudo ls /var/snap/lxd/common/lxd/snapshots  -lat
total 12
drwx--x--x 18 root root 4096 Dec 17 02:32 ..
drwx------  3 root root 4096 Feb 22  2020 .
lrwxrwxrwx  1 root root   50 Feb 22  2020 unifi -> /var/lib/lxd/storage-pools/default/snapshots/unifi
drwx------  4 root root 4096 Jul  4  2017 matrix

N.B.: The link target /var/lib/lxd/storage-pools/default/snapshots/unifi does not exist, but:

lfaraone@lithium:~$ sudo ls /var/snap/lxd/common/lxd/storage-pools/ -la
total 16
drwx--x--x  4 root root 4096 Apr 24  2018 .
drwx--x--x 18 root root 4096 Dec 17 02:32 ..
drwx--x--x  5 root root 4096 Dec 16 04:16 default
drwx--x--x  2 root root 4096 Apr 24  2018 mb
lfaraone@lithium:~$ sudo ls /var/snap/lxd/common/lxd/storage-pools/{mb,default} -la
/var/snap/lxd/common/lxd/storage-pools/default:
total 20
drwx--x--x  5 root root 4096 Dec 16 04:16 .
drwx--x--x  4 root root 4096 Apr 24  2018 ..
drwx--x--x 39 root root 4096 May 19  2019 containers
drwx------  4 root root 4096 Feb 22  2020 containers-snapshots
drwx------ 19 root root 4096 Dec 15 13:38 images

/var/snap/lxd/common/lxd/storage-pools/mb:
total 8
drwx--x--x 2 root root 4096 Apr 24  2018 .
drwx--x--x 4 root root 4096 Apr 24  2018 ..

I think we’ll first want to cleanup the ZFS state to bring things back to normal, then go cleanup /var/snap/lxd/common/lxd/snapshots to just contain symlinks to the relevant entries under /var/snap/lxd/common/lxd/storage-pools/ , that should make the upgrade logic work again

So /var/snap/lxd/common/lxd/snapshots/matrix should instead be a symlink to /var/lib/lxd/storage-pools/default/snapshots/matrix (does not exist, but matches other snapshot symlink) or /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/matrix/? I’ll note the first and last of those paths contain directories with different content:

lfaraone@lithium:~$ sudo ls /var/snap/lxd/common/lxd/snapshots/matrix  -lat
total 9
drwx------ 3 root   root   4096 Feb 22  2020 ..
drwx------ 4 root   root   4096 Jul  4  2017 .
drwx------ 2 root   root      2 Jul  4  2017 snap0
drwxr-xr-x 2 100000 100000    2 Jul  4  2017 snap2
lfaraone@lithium:~$ sudo ls -la /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/matrix/
total 24
drwx--x--x 6 root root 4096 Apr 24  2018 .
drwx------ 4 root root 4096 Feb 22  2020 ..
drwx--x--x 2 root root 4096 Jul  4  2017 snap0
drwx--x--x 2 root root 4096 Jul  4  2017 snap1
drwx--x--x 2 root root 4096 Jul  4  2017 snap2
drwx------ 2 root root 4096 Apr 24  2018 snap3

Can you show zfs list -t all?

Sure. I’ve removed some output that contained private names / were unrelated to lxd.

NAME                                                                                                USED  AVAIL  REFER  MOUNTPOINT
tank                                                                                               2.78T   751G   523G  /tank
tank/archive                                                                                       1.99T   751G  1.99T  /srv/mirror
tank/libvirt                                                                                       82.0G   751G   128K  /tank-bogus/libvirt
tank/libvirt/fedora                                                                                20.6G   766G  5.57G  -
[redacted]
tank/lxd                                                                                            189G   751G   128K  none
tank/lxd/containers                                                                                 171G   751G   128K  none
tank/lxd/containers/amusing-emu                                                                     223M   751G   898M  /var/snap/lxd/common/lxd/storage-pools/default/containers/amusing-emu
[redacted]
tank/lxd/containers/landscape                                                                      2.76G   751G  3.29G  /var/snap/lxd/common/lxd/storage-pools/default/containers/landscape
tank/lxd/containers/lxc-ci                                                                          122M   751G   817M  /var/snap/lxd/common/lxd/storage-pools/default/containers/lxc-ci
tank/lxd/containers/matrix                                                                         14.0G   751G  7.37G  /var/snap/lxd/common/lxd/storage-pools/default/containers/hdd-matrix
tank/lxd/containers/matrix@snapshot-snap0                                                          2.26G      -  3.78G  -
tank/lxd/containers/matrix@snapshot-snap1                                                           568M      -  3.94G  -
tank/lxd/containers/matrix@snapshot-snap2                                                           485M      -  3.91G  -
tank/lxd/containers/matrix@snapshot-snap3                                                          1.61G      -  4.63G  -
tank/lxd/containers/mediawiki                                                                      13.4G   751G  13.6G  /var/snap/lxd/common/lxd/storage-pools/default/containers/mediawiki
tank/lxd/containers/mediawiki-lxc                                                                  1.07G   751G  1.44G  
[redacted]
tank/lxd/containers/unifi                                                                          12.6G   751G  10.3G  /var/snap/lxd/common/lxd/storage-pools/default/containers/unifi
tank/lxd/containers/unifi@snapshot-2020-02-22-before-bionic                                        2.28G      -  6.64G  -
tank/lxd/containers/vcs                                                                             589M   751G   943M  /var/snap/lxd/common/lxd/storage-pools/default/containers/vcs
tank/lxd/containers/wildduck                                                                       2.59G   751G  3.22G  /var/snap/lxd/common/lxd/storage-pools/default/containers/wildduck
tank/lxd/deleted                                                                                   11.9G   751G   128K  none
tank/lxd/deleted/images                                                                            11.9G   751G   128K  none
tank/lxd/deleted/images/069b95ed3a60645ee1905b7625a468d1357f00bd61bf096fc597063c6ed42cf1            706M   751G   706M  none
tank/lxd/deleted/images/069b95ed3a60645ee1905b7625a468d1357f00bd61bf096fc597063c6ed42cf1@readonly     0B      -   706M  -
tank/lxd/deleted/images/086059fbf2aceafe5e539df4893ef8eb4856ce23ba520634e55c8fd22c09fff0            796M   751G   796M  none
tank/lxd/deleted/images/086059fbf2aceafe5e539df4893ef8eb4856ce23ba520634e55c8fd22c09fff0@readonly     0B      -   796M  -
tank/lxd/deleted/images/25a8bced78e47ab44d3633730a3ac3cdc3b70b6a5dd947bbdba1d2abbbee851a            406M   751G   406M  none
tank/lxd/deleted/images/25a8bced78e47ab44d3633730a3ac3cdc3b70b6a5dd947bbdba1d2abbbee851a@readonly     0B      -   406M  -
tank/lxd/deleted/images/2a8f52751e524f008a49c3b64f591311cc12e9361feee643caf98eaa9e8c3bd9            319M   751G   319M  none
tank/lxd/deleted/images/2a8f52751e524f008a49c3b64f591311cc12e9361feee643caf98eaa9e8c3bd9@readonly     0B      -   319M  -
tank/lxd/deleted/images/3e50ba589426c21f26370e2f949f30210f2d0419fbb9d4d4a0f860a035373353            798M   751G   798M  none
tank/lxd/deleted/images/3e50ba589426c21f26370e2f949f30210f2d0419fbb9d4d4a0f860a035373353@readonly     0B      -   798M  -
tank/lxd/deleted/images/489c3fa793c4a903c772099e71607606cc816f1fd3cc36e155bf1f06cd073684            325M   751G   325M  none
tank/lxd/deleted/images/489c3fa793c4a903c772099e71607606cc816f1fd3cc36e155bf1f06cd073684@readonly     0B      -   325M  -
tank/lxd/deleted/images/51a86a09a79d1c88c1f9956c10a61c0e2be56a3a50a12cd6749fa47d987c2cba            778M   751G   778M  none
tank/lxd/deleted/images/51a86a09a79d1c88c1f9956c10a61c0e2be56a3a50a12cd6749fa47d987c2cba@readonly     0B      -   778M  -
tank/lxd/deleted/images/5f0116a695ebbe22d6def3a9a97faad31710b1df592a12f616ca17e710681b94            282M   751G   282M  none
tank/lxd/deleted/images/5f0116a695ebbe22d6def3a9a97faad31710b1df592a12f616ca17e710681b94@readonly     0B      -   282M  -
tank/lxd/deleted/images/6442e2ab3364de369a106179ebb91c8129dd12069e866d57935767d7b9f69d99            272M   751G   272M  none
tank/lxd/deleted/images/6442e2ab3364de369a106179ebb91c8129dd12069e866d57935767d7b9f69d99@readonly     0B      -   272M  -
tank/lxd/deleted/images/725b9d539a4aef7d4405b09e831f741b9f8ef38caa067fffadf479e44b8bd22a            705M   751G   705M  none
tank/lxd/deleted/images/725b9d539a4aef7d4405b09e831f741b9f8ef38caa067fffadf479e44b8bd22a@readonly     0B      -   705M  -
tank/lxd/deleted/images/74a4918048777ca9fee9c0c02a9089134e28fff3f9c6e39423c8e0273d6c9b74            774M   751G   774M  none
tank/lxd/deleted/images/74a4918048777ca9fee9c0c02a9089134e28fff3f9c6e39423c8e0273d6c9b74@readonly     0B      -   774M  -
tank/lxd/deleted/images/764d358cdc07598355c303e72bfce96b1e14c847fe0b23225f99b6a72d5897fa            410M   751G   410M  none
tank/lxd/deleted/images/764d358cdc07598355c303e72bfce96b1e14c847fe0b23225f99b6a72d5897fa@readonly     0B      -   410M  -
tank/lxd/deleted/images/7a7ff654cbd8f5f09bec03aa19d8d7d92649127d18659036a963b1ea63f90d25            693M   751G   693M  none
tank/lxd/deleted/images/7a7ff654cbd8f5f09bec03aa19d8d7d92649127d18659036a963b1ea63f90d25@readonly     0B      -   693M  -
tank/lxd/deleted/images/872efe86fa5f08263fbde7a536b7bd02c5ac49eb4f8af7adbad19269e613b669            340M   751G   340M  none
tank/lxd/deleted/images/872efe86fa5f08263fbde7a536b7bd02c5ac49eb4f8af7adbad19269e613b669@readonly     0B      -   340M  -
tank/lxd/deleted/images/8fa08537ae51c880966626561987153e72d073cbe19dfe5abc062713d929254d            692M   751G   692M  none
tank/lxd/deleted/images/8fa08537ae51c880966626561987153e72d073cbe19dfe5abc062713d929254d@readonly     0B      -   692M  -
tank/lxd/deleted/images/b9cba741ef621f14e6e8c306c7f87637a59d7d09c1a3ec8ef717ab0d6e43430a            777M   751G   777M  none
tank/lxd/deleted/images/b9cba741ef621f14e6e8c306c7f87637a59d7d09c1a3ec8ef717ab0d6e43430a@readonly     0B      -   777M  -
tank/lxd/deleted/images/d0c72d121ba6674b6f34aa6ff247588033098627122b25d9bfec73a8303a2c3b            775M   751G   775M  none
tank/lxd/deleted/images/d0c72d121ba6674b6f34aa6ff247588033098627122b25d9bfec73a8303a2c3b@readonly     0B      -   775M  -
tank/lxd/deleted/images/da5746c04d9302a6804eda5b6fec4b654b78ae8a7adce9e4d1b6e71e4cc5a6b8            706M   751G   706M  none
tank/lxd/deleted/images/da5746c04d9302a6804eda5b6fec4b654b78ae8a7adce9e4d1b6e71e4cc5a6b8@readonly     0B      -   706M  -
tank/lxd/deleted/images/e6030cb0bd86e420056d567d541ed0939120af9ba6cf5a3506f48e2fa3b52dac            892M   751G   892M  none
tank/lxd/deleted/images/e6030cb0bd86e420056d567d541ed0939120af9ba6cf5a3506f48e2fa3b52dac@readonly     0B      -   892M  -
tank/lxd/deleted/images/f32f9de84a9e70b23f128f909f72ba484bc9ea70c69316ea5e32fb3c11282a34            705M   751G   705M  none
tank/lxd/deleted/images/f32f9de84a9e70b23f128f909f72ba484bc9ea70c69316ea5e32fb3c11282a34@readonly     0B      -   705M  -
tank/lxd/images                                                                                    6.14G   751G   128K  none
tank/lxd/images/0ba0194d3ad31e56309dca3b748bf0a8b96e652806729485def1577cb6b9de6d                    368M   751G   368M  none
tank/lxd/images/0ba0194d3ad31e56309dca3b748bf0a8b96e652806729485def1577cb6b9de6d@readonly             0B      -   368M  -
tank/lxd/images/0c99deeabeab64263a098296ded81efe87316631e3f0cec5792f458df474530d                    370M   751G   370M  none
tank/lxd/images/0c99deeabeab64263a098296ded81efe87316631e3f0cec5792f458df474530d@readonly             0B      -   370M  -
tank/lxd/images/2bf34c46293245fde1c1a5d0b9b07e7be50642d75285511fd5bcd76403e9c57f                    319M   751G   319M  none
tank/lxd/images/2bf34c46293245fde1c1a5d0b9b07e7be50642d75285511fd5bcd76403e9c57f@readonly             0B      -   319M  -
tank/lxd/images/58d840230b2242f4547f3adb2fcc9e972e719a62b98c02df800d4c638db8c502                    325M   751G   325M  none
tank/lxd/images/58d840230b2242f4547f3adb2fcc9e972e719a62b98c02df800d4c638db8c502@readonly             0B      -   325M  -
tank/lxd/images/68ddd0ffc223d04359cee1a46b0dff2d0609594959d8a3d719316d7a0d60b661                    295M   751G   295M  /var/snap/lxd/common/lxd/storage-pools/default/images/039261832
tank/lxd/images/7e8ee5ffd28732359c2ca5530d435ba0e8c4ffd25041788e755eb643c345b820                    284M   751G   284M  none
tank/lxd/images/7e8ee5ffd28732359c2ca5530d435ba0e8c4ffd25041788e755eb643c345b820@readonly             0B      -   284M  -
tank/lxd/images/97ca81741254148eae6a20e1d8f087d90a2bbf1b6c6116459bcdbe5dc5148ce3                    707M   751G   707M  none
tank/lxd/images/97ca81741254148eae6a20e1d8f087d90a2bbf1b6c6116459bcdbe5dc5148ce3@readonly             0B      -   707M  -
tank/lxd/images/be4aa8e56eab681fac6553b48ce19d7f34833accc2c8ae65f140a603b8369a1d                    864M   751G   864M  none
tank/lxd/images/be4aa8e56eab681fac6553b48ce19d7f34833accc2c8ae65f140a603b8369a1d@readonly             0B      -   864M  -
tank/lxd/images/bf58aeb73e909f0f5c47962d15e69cdd20c349bd8367b55e262df8d90d06ca26                    128K   751G   128K  /var/snap/lxd/common/lxd/storage-pools/default/images/151614147
tank/lxd/images/cc6879bc1091ef9a15e125312f799b6b4a66a94113c58521aeaebbf0456539dd                    128K   751G   128K  /var/snap/lxd/common/lxd/storage-pools/default/images/431646368
tank/lxd/images/cd3eb1babe6c3baa56b6702a4fca1699a3d7a613df3c98d4c649509525ad1949                    295M   751G   295M  none
tank/lxd/images/cd3eb1babe6c3baa56b6702a4fca1699a3d7a613df3c98d4c649509525ad1949@readonly             0B      -   295M  -
tank/lxd/images/de03ec8f36f75815b6b6c6dcae2dc0c7b09079b035a27c46ff854b0977f9139b                    753M   751G   753M  none
tank/lxd/images/de03ec8f36f75815b6b6c6dcae2dc0c7b09079b035a27c46ff854b0977f9139b@readonly             0B      -   753M  -
tank/lxd/images/e56517227c702301ec690bea51ae82be05eec3825b223d4c933fd4ad8c7cff03                    308M   751G   308M  none
tank/lxd/images/e56517227c702301ec690bea51ae82be05eec3825b223d4c933fd4ad8c7cff03@readonly             0B      -   308M  -
tank/lxd/images/e5672d2ed5e7c85935f3d887ce7817067296554df1f7421855d1b8df13abbedf                    325M   751G   325M  none
tank/lxd/images/e5672d2ed5e7c85935f3d887ce7817067296554df1f7421855d1b8df13abbedf@readonly             0B      -   325M  -
tank/lxd/images/ea0f73f7dd8025b28f7e260f588761e2d43b657c6245a10b4e3b9f1a9dd0308e                    295M   751G   295M  /var/snap/lxd/common/lxd/storage-pools/default/images/245542578
tank/lxd/images/f8c28a24b348313d5b842edb25520ae87e34729728ddee5b8001c639fecc97d2                    776M   751G   776M  none
tank/lxd/images/f8c28a24b348313d5b842edb25520ae87e34729728ddee5b8001c639fecc97d2@readonly             0B      -   776M  -
tank/lxd/snapshots                                                                                  821K   751G   128K  none
tank/lxd/snapshots/matrix                                                                           565K   751G   128K  none
tank/lxd/snapshots/matrix/snap0                                                                     181K   751G   224K  /var/snap/lxd/common/lxd/snapshots/matrix/snap0
tank/lxd/snapshots/matrix/snap2                                                                     256K   751G   362K  /var/snap/lxd/common/lxd/snapshots/matrix/snap2
tank/lxd/snapshots/unifi                                                                            128K   751G   128K  none
tank/varx                                                                                          8.14G   751G   128K  /tank-bogus/varx
tank/varx/uvtool                                                                                   8.14G   751G  8.14G  /var/lib/uvtool

You’re kinda halfway through that update so that would explain the existence of both snapshots and containers-snapshots in the storage pool directory.

If your container exists in either of those, symlink to that, if it doesn’t exist in either and has some snapshots in zfs, then create the directory under containers-snapshots and create a sub-directory for each snapshot, then make the symlink that points to that.

Thanks so much! It existed in containers-snapshots, and once I created the appropriate symlink, LXD was able to start successfully.

root@lithium:/var/snap/lxd/common/lxd/snapshots# ls -lat
total 12
drwx--x--x 18 root root 4096 Dec 17 02:32 ..
drwx------  3 root root 4096 Feb 22  2020 .
lrwxrwxrwx  1 root root   50 Feb 22  2020 unifi -> /var/lib/lxd/storage-pools/default/snapshots/unifi
drwx------  4 root root 4096 Jul  4  2017 matrix
root@lithium:/var/snap/lxd/common/lxd/snapshots# mv matrix ../matrix-save
root@lithium:/var/snap/lxd/common/lxd/snapshots# ln -s /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/matrix matrix

Excellent!

Sorry for the bumpy upgrade.