Troubleshooting UNAVAILABLE storage

I see this when I run incus storage list.

$ incus storage list
+---------------+--------+--------------------------------------------+-------------+---------+-------------+
|     NAME      | DRIVER |                   SOURCE                   | DESCRIPTION | USED BY |    STATE    |
+---------------+--------+--------------------------------------------+-------------+---------+-------------+
| default2      | zfs    | /var/snap/lxd/common/lxd/disks/default.img |             | 9       | UNAVAILABLE |
+---------------+--------+--------------------------------------------+-------------+---------+-------------+
| incus-default | zfs    | /var/lib/incus/disks/incus-default.img     |             | 5       | CREATED     |
+---------------+--------+--------------------------------------------+-------------+---------+-------------+

The default2 disk is definitely an old disk imported from lxd. But, the file exists:

sudo ls -lah /var/snap/lxd/common/lxd/disks/default.img
[sudo] password for xrd: 
-rw------- 1 root root 100G Nov 22 08:35 /var/snap/lxd/common/lxd/disks/default.img

zpool and zfs don’t show it.

$ zpool list
NAME            SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
incus-default  29.5G  7.60G  21.9G        -         -    18%    25%  1.00x    ONLINE  -
$ zfs list | grep default2
<empty...>

I’ve tried a combination of zpool import and incus admin recover but nothing seems to work.

$ incus admin recover
This server currently has the following storage pools:
 - default2 (backend="zfs", source="/var/snap/lxd/common/lxd/disks/default.img")
 - incus-default (backend="zfs", source="/var/lib/incus/disks/incus-default.img")
Would you like to recover another storage pool? (yes/no) [default=no]: no
The recovery process will be scanning the following storage pools:
 - EXISTING: "default2" (backend="zfs", source="/var/snap/lxd/common/lxd/disks/default.img")
 - EXISTING: "incus-default" (backend="zfs", source="/var/lib/incus/disks/incus-default.img")
Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]: 
Scanning for unknown volumes...
Error: Failed validation request: Failed mounting pool "default2": Failed to run: zpool import -f -d /var/lib/incus/disks default2: exit status 1 (cannot import 'default2': no such pool available)

$ zpool import -f -d /var/snap/lxd/common default2
cannot import 'default2': no such pool available

When you did the migration, I suppose you opted not to remove the old storage pool.
Can you check whether the old storage pool is used at all?
The following command uses the b flag that shows the storage pool for each instance.

incus list -c ns4tb

Can you show the exact error message when you try to launch a container on the old storage pool?

incus launch images:alpine/edge mycontainer --storage default2

Hi @simos , thanks for your help.

Yes, the storage pool is used by a few machines, sadly.

$ incus launch images:alpine/edge testcontainer --storage default2
Launching testcontainer
Error: Failed instance creation: Failed creating instance from image: Storage pool is unavailable on this server

This is basically what happens when I try to launch one of my VMs on that storage pool:

$ incus start community-public-do-vm
Error: Storage pool "default2" unavailable on this server

Also, I should say, I did have this pool working correctly until a reboot the other day. So, the initial migration from lxd did work.

There was a USB drive which was not plugged in correctly and I suspect that had a cascading effect where other things failed, and all the mounts did not succeed. But, I am having trouble trying to figure out how to resolve it all.

For more context, this storage pool was from an old installation and on a different drive. I moved it into the main SSD drive, and perhaps I used the wrong name? It was originally default, but I think I had to rename it to default2 to get things working. But, this could be memory being faulty.

Continuing to look, not sure if this indicates the filesystem went bad.

$ sudo mount  /var/snap/lxd/common/lxd/disks/default.img /mnt/zpool/
mount: /mnt/zpool: unknown filesystem type 'zfs_member'.
       dmesg(1) may have more information after failed mount system call.
sudo mount -t zfs /var/snap/lxd/common/lxd/disks/default.img /mnt/zpool/
filesystem 'default' cannot be mounted, unable to open the dataset
$ sudo mount  /var/snap/lxd/common/lxd/disks/default.img /mnt/zpool/
mount: /mnt/zpool: unknown filesystem type 'zfs_member'.
       dmesg(1) may have more information after failed mount system call.
$ zdb -l /var/snap/lxd/common/lxd/disks/default.img 
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'default'
    state: 2
    txg: 672799
    pool_guid: 8014720309659989416
    errata: 0
    hostid: 1982906199
    hostname: 'biggpu'
    top_guid: 2738119307724156891
    guid: 2738119307724156891
    vdev_children: 1
    vdev_tree:
        type: 'file'
        id: 0
        guid: 2738119307724156891
        path: '/var/snap/lxd/common/lxd/disks/default.img'
        metaslab_array: 64
        metaslab_shift: 29
        ashift: 9
        asize: 107369463808
        is_log: 0
        DTL: 5108
        create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 0 1 2 3