Ubuntu 16.04 -> 18.04 -> 20.04 lxd.migration failed because of non existent source LXD

devzero · December 21, 2022, 9:22am

Hi,

I have upraded an old host from 16 to 18 to 20.04 which worked, however the migration script is not working, because the former LXD is uninstalled because it was found as the ubuntu 16.04.

Obviously I should have done a dust-upgrade at 18.04 and check if the LXD is upgraded proper in that state, however I relied on the do-release-upgrade tool.

dpkg is now showing the following:

dpkg -l | grep lxc
ri  liblxc-common                          1:4.0.12-0ubuntu1~20.04.1         amd64        Linux Containers userspace tools (common tools)
ii  liblxc1                                1:4.0.12-0ubuntu1~20.04.1         amd64        Linux Containers userspace tools (library)
ii  lxc                                    1:4.0.12-0ubuntu1~20.04.1         all          Transitional package - lxc -> lxc-utils
rc  lxc-common                             2.0.11-0ubuntu1~16.04.3           amd64        Linux Containers userspace tools (common tools)
ii  lxc-utils                              1:4.0.12-0ubuntu1~20.04.1         amd64        Linux Containers userspace tools
ii  lxcfs                                  4.0.3-0ubuntu1                    amd64        FUSE based filesystem for LXC

and I suspect that at 18.04 I missed checking everything in detail.

The question is now, how do I get a working source LXD to get the containers migrated to snap, or is there a manual how to migrate them by hand if the prior version is gone?

stgraber · December 23, 2022, 2:47am

Can you show:

dpkg -l | grep lxd
snap list
ls -lh /var/lib/lxd/
ls -lh /var/snap/lxd/common/lxd/

devzero · December 23, 2022, 8:04am

Hi, here is the output:

dpkg -l | grep lxd
ii  lxd                                    1:0.10                            all          Transitional package - lxd -> snap (lxd)
ii  lxd-client                             1:0.10                            all          Transitional package - lxd-client -> LXD snap

snap list
Name    Version        Rev    Tracking       Publisher   Notes
core20  20221123       1738   latest/stable  canonical✓  base
lxd     4.0.9-a29c6f1  24061  4.0/stable/…   canonical✓  -
snapd   2.57.6         17883  latest/stable  canonical✓  snapd

ls -lh /var/lib/lxd/
total 4.0K
drwxr-xr-x 3 root root 4.0K Dec 22 15:07 storage-pools

ls -lh /var/snap/lxd/common/lxd/
total 68K
drwx------ 2 root root 4.0K Dec 20 10:45 backups
drwx------ 2 root root 4.0K Dec 20 10:45 cache
drwx--x--x 2 root root 4.0K Dec 20 10:45 containers
drwx------ 3 root root 4.0K Dec 21 09:22 database
drwx--x--x 2 root root 4.0K Dec 21 09:22 devices
drwxr-xr-x 2 root root 4.0K Dec 20 10:45 devlxd
drwx------ 2 root root 4.0K Dec 20 10:45 disks
drwx------ 2 root root 4.0K Dec 20 10:45 images
drwx------ 2 root root 4.0K Dec 22 14:56 logs
drwx--x--x 2 root root 4.0K Dec 22 14:57 networks
srwx------ 1 root root    0 Dec 21 09:22 seccomp.socket
drwx------ 4 root root 4.0K Dec 20 10:45 security
-rw-r--r-- 1 root root  782 Dec 20 10:45 server.crt
-rw------- 1 root root  288 Dec 20 10:45 server.key
lrwxrwxrwx 1 root root   39 Dec 21 09:22 shmounts -> /var/snap/lxd/common/shmounts/instances
drwx------ 2 root root 4.0K Dec 20 10:45 snapshots
drwx--x--x 4 root root 4.0K Dec 22 15:27 storage-pools
srw-rw---- 1 root lxd     0 Dec 21 09:18 unix.socket
drwx--x--x 2 root root 4.0K Dec 20 10:45 virtual-machines
drwx------ 2 root root 4.0K Dec 20 10:45 virtual-machines-snapshots

The zpool is still here and the containers are still in the zpool, however not migrated to the snap LXD

stgraber · December 24, 2022, 1:00am

Can you do /snap/bin/lxd sql global .dump?

The filesystem structure shows nothing left in the old location, so I’m hoping that most data did end up being transferred somehow.

Also, can you show ls -lh /var/snap/lxd/common/lxd/containers?

devzero · December 26, 2022, 11:19am

The output of the SQL dump is:

PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE schema (
    id         INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    version    INTEGER NOT NULL,
    updated_at DATETIME NOT NULL,
    UNIQUE (version)
);
INSERT INTO schema VALUES(1,30,1671529543);
CREATE TABLE "images" (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    fingerprint TEXT NOT NULL,
    filename TEXT NOT NULL,
    size INTEGER NOT NULL,
    public INTEGER NOT NULL DEFAULT 0,
    architecture INTEGER NOT NULL,
    creation_date DATETIME,
    expiry_date DATETIME,
    upload_date DATETIME NOT NULL,
    cached INTEGER NOT NULL DEFAULT 0,
    last_use_date DATETIME,
    auto_update INTEGER NOT NULL DEFAULT 0,
    project_id INTEGER NOT NULL,
    type INTEGER NOT NULL DEFAULT 0,
    UNIQUE (project_id, fingerprint),
    FOREIGN KEY (project_id) REFERENCES projects (id) ON DELETE CASCADE
);
CREATE TABLE "images_aliases" (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    name TEXT NOT NULL,
    image_id INTEGER NOT NULL,
    description TEXT,
    project_id INTEGER NOT NULL,
    UNIQUE (project_id, name),
    FOREIGN KEY (image_id) REFERENCES images (id) ON DELETE CASCADE,
    FOREIGN KEY (project_id) REFERENCES projects (id) ON DELETE CASCADE
);
CREATE TABLE "instances" (
    id INTEGER primary key AUTOINCREMENT NOT NULL,
    node_id INTEGER NOT NULL,
    name TEXT NOT NULL,
    architecture INTEGER NOT NULL,
    type INTEGER NOT NULL,
    ephemeral INTEGER NOT NULL DEFAULT 0,
    creation_date DATETIME NOT NULL DEFAULT 0,
    stateful INTEGER NOT NULL DEFAULT 0,
    last_use_date DATETIME,
    description TEXT,
    project_id INTEGER NOT NULL,
    expiry_date DATETIME,
    UNIQUE (project_id, name),
    FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE,
    FOREIGN KEY (project_id) REFERENCES projects (id) ON DELETE CASCADE
);
CREATE TABLE "instances_backups" (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    instance_id INTEGER NOT NULL,
    name VARCHAR(255) NOT NULL,
    creation_date DATETIME,
    expiry_date DATETIME,
    container_only INTEGER NOT NULL default 0,
    optimized_storage INTEGER NOT NULL default 0,
    FOREIGN KEY (instance_id) REFERENCES "instances" (id) ON DELETE CASCADE,
    UNIQUE (instance_id, name)
);
CREATE TABLE "instances_config" (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    instance_id INTEGER NOT NULL,
    key TEXT NOT NULL,
    value TEXT,
    FOREIGN KEY (instance_id) REFERENCES "instances" (id) ON DELETE CASCADE,
    UNIQUE (instance_id, key)
);
CREATE TABLE "instances_devices" (
    id INTEGER primary key AUTOINCREMENT NOT NULL,
    instance_id INTEGER NOT NULL,
    name TEXT NOT NULL,
    type INTEGER NOT NULL default 0,
    FOREIGN KEY (instance_id) REFERENCES "instances" (id) ON DELETE CASCADE,
    UNIQUE (instance_id, name)
);
CREATE TABLE "instances_devices_config" (
    id INTEGER primary key AUTOINCREMENT NOT NULL,
    instance_device_id INTEGER NOT NULL,
    key TEXT NOT NULL,
    value TEXT,
    FOREIGN KEY (instance_device_id) REFERENCES "instances_devices" (id) ON DELETE CASCADE,
    UNIQUE (instance_device_id, key)
);
CREATE TABLE "instances_profiles" (
    id INTEGER primary key AUTOINCREMENT NOT NULL,
    instance_id INTEGER NOT NULL,
    profile_id INTEGER NOT NULL,
    apply_order INTEGER NOT NULL default 0,
    UNIQUE (instance_id, profile_id),
    FOREIGN KEY (instance_id) REFERENCES "instances"(id) ON DELETE CASCADE,
    FOREIGN KEY (profile_id) REFERENCES profiles(id) ON DELETE CASCADE
);
CREATE TABLE "operations" (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    uuid TEXT NOT NULL,
    node_id TEXT NOT NULL,
    type INTEGER NOT NULL DEFAULT 0,
    project_id INTEGER,
    UNIQUE (uuid),
    FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE,
    FOREIGN KEY (project_id) REFERENCES projects (id) ON DELETE CASCADE
);
CREATE TABLE "profiles" (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    name TEXT NOT NULL,
    description TEXT,
    project_id INTEGER NOT NULL,
    UNIQUE (project_id, name),
    FOREIGN KEY (project_id) REFERENCES projects (id) ON DELETE CASCADE
);
INSERT INTO "profiles" VALUES(1,'default','Default LXD profile',1);
CREATE TABLE "storage_volumes" (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    name TEXT NOT NULL,
    storage_pool_id INTEGER NOT NULL,
    node_id INTEGER NOT NULL,
    type INTEGER NOT NULL,
    description TEXT,
    project_id INTEGER NOT NULL,
    UNIQUE (storage_pool_id, node_id, project_id, name, type),
    FOREIGN KEY (storage_pool_id) REFERENCES storage_pools (id) ON DELETE CASCADE,
    FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE,
    FOREIGN KEY (project_id) REFERENCES projects (id) ON DELETE CASCADE
);
CREATE TABLE certificates (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    fingerprint TEXT NOT NULL,
    type INTEGER NOT NULL,
    name TEXT NOT NULL,
    certificate TEXT NOT NULL,
    UNIQUE (fingerprint)
);
CREATE TABLE config (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    key TEXT NOT NULL,
    value TEXT,
    UNIQUE (key)
);
CREATE TABLE images_nodes (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    image_id INTEGER NOT NULL,
    node_id INTEGER NOT NULL,
    UNIQUE (image_id, node_id),
    FOREIGN KEY (image_id) REFERENCES images (id) ON DELETE CASCADE,
    FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE
);
CREATE TABLE images_profiles (
    image_id INTEGER NOT NULL,
    profile_id INTEGER NOT NULL,
    FOREIGN KEY (image_id) REFERENCES images (id) ON DELETE CASCADE,
    FOREIGN KEY (profile_id) REFERENCES profiles (id) ON DELETE CASCADE,
    UNIQUE (image_id, profile_id)
);
CREATE TABLE images_properties (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    image_id INTEGER NOT NULL,
    type INTEGER NOT NULL,
    key TEXT NOT NULL,
    value TEXT,
    FOREIGN KEY (image_id) REFERENCES images (id) ON DELETE CASCADE
);
CREATE TABLE images_source (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    image_id INTEGER NOT NULL,
    server TEXT NOT NULL,
    protocol INTEGER NOT NULL,
    certificate TEXT NOT NULL,
    alias TEXT NOT NULL,
    FOREIGN KEY (image_id) REFERENCES images (id) ON DELETE CASCADE
);
CREATE TABLE instances_snapshots (
    id INTEGER primary key AUTOINCREMENT NOT NULL,
    instance_id INTEGER NOT NULL,
    name TEXT NOT NULL,
    creation_date DATETIME NOT NULL DEFAULT 0,
    stateful INTEGER NOT NULL DEFAULT 0,
    description TEXT,
    expiry_date DATETIME,
    UNIQUE (instance_id, name),
    FOREIGN KEY (instance_id) REFERENCES instances (id) ON DELETE CASCADE
);
CREATE TABLE instances_snapshots_config (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    instance_snapshot_id INTEGER NOT NULL,
    key TEXT NOT NULL,
    value TEXT,
    FOREIGN KEY (instance_snapshot_id) REFERENCES instances_snapshots (id) ON DELETE CASCADE,
    UNIQUE (instance_snapshot_id, key)
);
CREATE TABLE instances_snapshots_devices (
    id INTEGER primary key AUTOINCREMENT NOT NULL,
    instance_snapshot_id INTEGER NOT NULL,
    name TEXT NOT NULL,
    type INTEGER NOT NULL default 0,
    FOREIGN KEY (instance_snapshot_id) REFERENCES instances_snapshots (id) ON DELETE CASCADE,
    UNIQUE (instance_snapshot_id, name)
);
CREATE TABLE instances_snapshots_devices_config (
    id INTEGER primary key AUTOINCREMENT NOT NULL,
    instance_snapshot_device_id INTEGER NOT NULL,
    key TEXT NOT NULL,
    value TEXT,
    FOREIGN KEY (instance_snapshot_device_id) REFERENCES instances_snapshots_devices (id) ON DELETE CASCADE,
    UNIQUE (instance_snapshot_device_id, key)
);
CREATE TABLE networks (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    name TEXT NOT NULL,
    description TEXT,
    state INTEGER NOT NULL DEFAULT 0,
    UNIQUE (name)
);
CREATE TABLE networks_config (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    network_id INTEGER NOT NULL,
    node_id INTEGER,
    key TEXT NOT NULL,
    value TEXT,
    UNIQUE (network_id, node_id, key),
    FOREIGN KEY (network_id) REFERENCES networks (id) ON DELETE CASCADE,
    FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE
);
CREATE TABLE networks_nodes (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    network_id INTEGER NOT NULL,
    node_id INTEGER NOT NULL,
    UNIQUE (network_id, node_id),
    FOREIGN KEY (network_id) REFERENCES networks (id) ON DELETE CASCADE,
    FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE
);
CREATE TABLE nodes (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    name TEXT NOT NULL,
    description TEXT DEFAULT '',
    address TEXT NOT NULL,
    schema INTEGER NOT NULL,
    api_extensions INTEGER NOT NULL,
    heartbeat DATETIME DEFAULT CURRENT_TIMESTAMP,
    pending INTEGER NOT NULL DEFAULT 0,
    arch INTEGER NOT NULL DEFAULT 0 CHECK (arch > 0),
    UNIQUE (name),
    UNIQUE (address)
);
INSERT INTO nodes VALUES(1,'none','','0.0.0.0',30,220,1671529543,0,2);
CREATE TABLE nodes_roles (
    node_id INTEGER NOT NULL,
    role INTEGER NOT NULL,
    FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE,
    UNIQUE (node_id, role)
);
CREATE TABLE profiles_config (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    profile_id INTEGER NOT NULL,
    key TEXT NOT NULL,
    value TEXT,
    UNIQUE (profile_id, key),
    FOREIGN KEY (profile_id) REFERENCES profiles(id) ON DELETE CASCADE
);
CREATE TABLE profiles_devices (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    profile_id INTEGER NOT NULL,
    name TEXT NOT NULL,
    type INTEGER NOT NULL default 0,
    UNIQUE (profile_id, name),
    FOREIGN KEY (profile_id) REFERENCES profiles (id) ON DELETE CASCADE
);
CREATE TABLE profiles_devices_config (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    profile_device_id INTEGER NOT NULL,
    key TEXT NOT NULL,
    value TEXT,
    UNIQUE (profile_device_id, key),
    FOREIGN KEY (profile_device_id) REFERENCES profiles_devices (id) ON DELETE CASCADE
);
CREATE TABLE projects (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    name TEXT NOT NULL,
    description TEXT,
    UNIQUE (name)
);
INSERT INTO projects VALUES(1,'default','Default LXD project');
CREATE TABLE projects_config (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    project_id INTEGER NOT NULL,
    key TEXT NOT NULL,
    value TEXT,
    FOREIGN KEY (project_id) REFERENCES projects (id) ON DELETE CASCADE,
    UNIQUE (project_id, key)
);
INSERT INTO projects_config VALUES(1,1,'features.images','true');
INSERT INTO projects_config VALUES(2,1,'features.profiles','true');
INSERT INTO projects_config VALUES(3,1,'features.storage.volumes','true');
CREATE TABLE storage_pools (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    name TEXT NOT NULL,
    driver TEXT NOT NULL,
    description TEXT,
    state INTEGER NOT NULL DEFAULT 0,
    UNIQUE (name)
);
CREATE TABLE storage_pools_config (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    storage_pool_id INTEGER NOT NULL,
    node_id INTEGER,
    key TEXT NOT NULL,
    value TEXT,
    UNIQUE (storage_pool_id, node_id, key),
    FOREIGN KEY (storage_pool_id) REFERENCES storage_pools (id) ON DELETE CASCADE,
    FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE
);
CREATE TABLE storage_pools_nodes (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    storage_pool_id INTEGER NOT NULL,
    node_id INTEGER NOT NULL,
    UNIQUE (storage_pool_id, node_id),
    FOREIGN KEY (storage_pool_id) REFERENCES storage_pools (id) ON DELETE CASCADE,
    FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE
);
CREATE TABLE storage_volumes_config (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    storage_volume_id INTEGER NOT NULL,
    key TEXT NOT NULL,
    value TEXT,
    UNIQUE (storage_volume_id, key),
    FOREIGN KEY (storage_volume_id) REFERENCES storage_volumes (id) ON DELETE CASCADE
);
CREATE TABLE storage_volumes_snapshots (
    id INTEGER NOT NULL,
    storage_volume_id INTEGER NOT NULL,
    name TEXT NOT NULL,
    description TEXT,
    expiry_date DATETIME,
    UNIQUE (id),
    UNIQUE (storage_volume_id, name),
    FOREIGN KEY (storage_volume_id) REFERENCES storage_volumes (id) ON DELETE CASCADE
);
CREATE TABLE storage_volumes_snapshots_config (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    storage_volume_snapshot_id INTEGER NOT NULL,
    key TEXT NOT NULL,
    value TEXT,
    FOREIGN KEY (storage_volume_snapshot_id) REFERENCES storage_volumes_snapshots (id) ON DELETE CASCADE,
    UNIQUE (storage_volume_snapshot_id, key)
);
DELETE FROM sqlite_sequence;
INSERT INTO sqlite_sequence VALUES('schema',1);
INSERT INTO sqlite_sequence VALUES('nodes',1);
INSERT INTO sqlite_sequence VALUES('projects',1);
INSERT INTO sqlite_sequence VALUES('projects_config',3);
INSERT INTO sqlite_sequence VALUES('profiles',1);
INSERT INTO sqlite_sequence VALUES('operations',206);
INSERT INTO sqlite_sequence VALUES('networks',1);
INSERT INTO sqlite_sequence VALUES('networks_nodes',1);
INSERT INTO sqlite_sequence VALUES('networks_config',4);
INSERT INTO sqlite_sequence VALUES('storage_pools',1);
INSERT INTO sqlite_sequence VALUES('storage_pools_nodes',1);
INSERT INTO sqlite_sequence VALUES('storage_pools_config',2);
COMMIT;

the ls shows:

ls -lh /var/snap/lxd/common/lxd/containers

total 0

however zfs list is showing this:

zfs list
NAME                                                                                  USED  AVAIL     REFER  MOUNTPOINT
lxd                                                                                   232G  1.04T       96K  none
lxd/containers                                                                        225G  1.04T       96K  none
lxd/containers/ctyzsrvct101                                                           35.6G  1.04T     35.6G  /var/lib/lxd/storage-pools/default/containers/ctyzsrvct101
lxd/containers/ctyzsrvct102NAME38                                                      3.87G  1.04T     3.87G  /var/lib/lxd/storage-pools/default/containers/ctyzsrvct102srv38
lxd/containers/ctyzsrvct104                                                           7.41G  56.6G     7.62G  /var/lib/lxd/storage-pools/default/containers/ctyzsrvct104
lxd/containers/ctyzsrvct104Dead                                                       2.55G  1.04T     2.55G  /var/lib/lxd/storage-pools/default/containers/ctyzsrvct104Dead
lxd/containers/ctyzsrvct201                                                           93.6G  1.04T     93.6G  /var/lib/lxd/storage-pools/default/containers/ctyzsrvct201
lxd/containers/ctyzsrvct203                                                            714M  1.04T      714M  /var/lib/lxd/storage-pools/default/containers/ctyzsrvct203
lxd/containers/ctyzsrvct205                                                            717M  1.04T      717M  /var/lib/lxd/storage-pools/default/containers/ctyzsrvct205
lxd/containers/ctyzanbct111                                                           7.63G  24.4G     7.87G  /var/lib/lxd/storage-pools/default/containers/ctyzanbct111
lxd/containers/ctyzanbct210                                                           2.36G  1.04T     2.36G  /var/lib/lxd/storage-pools/default/containers/ctyzanbct210
lxd/containers/ctyzansct111                                                           1.20G  1.04T     1.20G  /var/lib/lxd/storage-pools/default/containers/ctyzansct111
lxd/containers/ctyzasbct111                                                           7.68G  24.3G     7.92G  /var/lib/lxd/storage-pools/default/containers/ctyzasbct111
lxd/containers/ctyzasbct210                                                           6.22G  1.04T     7.20G  /var/lib/lxd/storage-pools/default/containers/ctyzasbct210
lxd/containers/ctyzaxmct111                                                           1.81G  1.04T     1.81G  /var/lib/lxd/storage-pools/default/containers/ctyzaxmct111
lxd/containers/ctyzaxmct210                                                           1.06G  1.04T     1.06G  /var/lib/lxd/storage-pools/default/containers/ctyzaxmct210
lxd/containers/ctyzaxtct111                                                           1.89G  30.1G     2.10G  /var/lib/lxd/storage-pools/default/containers/ctyzaxtct111
lxd/containers/ctyzaxtct210                                                            718M  1.04T      718M  /var/lib/lxd/storage-pools/default/containers/ctyzaxtct210
lxd/containers/ctyzgrafpe10                                                          2.38G  1.04T     2.52G  /var/lib/lxd/storage-pools/default/containers/ctyzgrafpe10
lxd/containers/ctyzgrape11                                                           6.77G  1.04T     7.39G  /var/lib/lxd/storage-pools/default/containers/ctyzgrape11
lxd/containers/ctyzgrape11-bak                                                       3.20M  1.04T      297M  /var/lib/lxd/storage-pools/default/containers/ctyzgrape11-bak
lxd/containers/ctyzgrdpe10-BAK                                                       31.8G  1.04T     31.8G  /var/lib/lxd/storage-pools/default/containers/ctyzgrdpe10-BAK
lxd/containers/ctyzlbxct111                                                            926M  1.04T      926M  /var/lib/lxd/storage-pools/default/containers/ctyzlbxct111
lxd/containers/ctyzlbxct113                                                           1.47G  1.04T     1.47G  /var/lib/lxd/storage-pools/default/containers/ctyzlbxct113
lxd/containers/ctyzlbxct210                                                            939M  1.04T      939M  /var/lib/lxd/storage-pools/default/containers/ctyzlbxct210
lxd/containers/ctyzlbxct212                                                           5.62G  1.04T     5.62G  /var/lib/lxd/storage-pools/default/containers/ctyzlbxct212

the Containers are still mountable with zfs mount.
If you do a lxd init with this storage set, you get the error that it is not empty.
So I am curious what the migration script is doing differently here.