Lxd-to-incus migration failed: Failed to initialize global database

Incus
1

Hi, I migrated another system from from LXD to Incus and it ran perfectly, but today I tried migrating and noted that my version of LXD was too new. I ran “snap refresh lxd --channel=5.21/stable” to downgrade and everything seemed ok, so ran lxd-to-incus which appeared to complete migrations, but failed to start incus.

The error in the log file (/var/log/incus/incusd.log) is:
level=error msg=“Failed to start the daemon” err=“Failed to initialize global database: Failed to prepare statements: "\nDELETE FROM instances WHERE project_id = (SELECT projects.id FROM projects WHERE projects.name = ?) AND name = ?\n": no such table: main.auth_groups_permissions”

2

Can you show the lxd-to-incus log, it should be in /var/log/.

We can’t look at the LXD code past 5.21 without risking legal complications which is why 5.21 is the last LXD release we support upgrading from. If the LXD team made any database change in releases after that, then the downgrade is unsafe and the database schema may be quite messed up.

3

Thanks for the quick reply. It took a few hours, but I’m up and running again. I moved the database files and Incus started creating a new blank database. I used the incus admin recover to restore my containers from storage, but I had to manually edit each backup.yaml to remove the lines that reference “volatile.uuid” before it would work. I had 1 container that refused to start with a container snapshot miscount, and figured out I just needed to delete the reference. I reattached the network bridge to each container and networking worked normally. I still need to recreate a profile, but that can wait.
I connected to a remote Incus instance and copied containers without issue and my apps all seem to be running normally, so it looks healthy.
I only wish I hadn’t put off migrating. Thanks again.

4

Here is the log. Everything seemed to be going well:

Source server: snap package
Target server: systemd
Source server paths: &{daemon:/var/snap/lxd/common/lxd logs:/var/snap/lxd/common/lxd/logs cache:/var/snap/lxd/common/lxd/cache}
Target server paths: &{daemon:/var/lib/incus logs:/var/log/incus cache:/var/cache/incus}
Rewrite SQL statements:
 - UPDATE profiles SET description='Default Incus profile' WHERE description='Default LXD profile';
 - UPDATE projects SET description='Default Incus project' WHERE description='Default LXD project';
 - DELETE FROM storage_volumes_config WHERE key='volatile.uuid';
 - DELETE FROM storage_volumes_snapshots_config WHERE key='volatile.uuid';
 - CREATE TABLE certificates (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    fingerprint TEXT NOT NULL,
    type INTEGER NOT NULL,
    name TEXT NOT NULL,
    certificate TEXT NOT NULL,
    restricted INTEGER NOT NULL DEFAULT 0,
    UNIQUE (fingerprint)
);
CREATE TABLE "certificates_projects" (
    certificate_id INTEGER NOT NULL,
    project_id INTEGER NOT NULL,
    FOREIGN KEY (certificate_id) REFERENCES certificates (id) ON DELETE CASCADE,
    FOREIGN KEY (project_id) REFERENCES "projects" (id) ON DELETE CASCADE,
    UNIQUE (certificate_id, project_id)
);
 - DELETE FROM schema WHERE version < 73;
UPDATE schema SET version=69 WHERE version=73;
 - INSERT INTO certificates (id, fingerprint, type, name, certificate, restricted) SELECT id, identifier, 1, name, json_extract(metadata, "$.cert"), 1 FROM identities WHERE type=1;
INSERT INTO certificates (id, fingerprint, type, name, certificate, restricted) SELECT id, identifier, 1, name, json_extract(metadata, "$.cert"), 0 FROM identities WHERE type=2;
INSERT INTO certificates (id, fingerprint, type, name, certificate, restricted) SELECT id, identifier, 2, name, json_extract(metadata, "$.cert"), 0 FROM identities WHERE type=3;
INSERT INTO certificates (id, fingerprint, type, name, certificate, restricted) SELECT id, identifier, 3, name, json_extract(metadata, "$.cert"), 1 FROM identities WHERE type=4;
INSERT INTO certificates (id, fingerprint, type, name, certificate, restricted) SELECT id, identifier, 3, name, json_extract(metadata, "$.cert"), 0 FROM identities WHERE type=6;
INSERT INTO certificates_projects (certificate_id, project_id) SELECT identity_id, project_id FROM identities_projects;
 - DROP TRIGGER on_auth_group_delete;
DROP TRIGGER on_cluster_group_delete;
DROP TRIGGER on_identity_delete;
DROP TRIGGER on_identity_provider_group_delete;
DROP TRIGGER on_image_alias_delete;
DROP TRIGGER on_image_delete;
DROP TRIGGER on_instance_backup_delete;
DROP TRIGGER on_instance_delete;
DROP TRIGGER on_instance_snaphot_delete;
DROP TRIGGER on_network_acl_delete;
DROP TRIGGER on_network_delete;
DROP TRIGGER on_network_zone_delete;
DROP TRIGGER on_node_delete;
DROP TRIGGER on_operation_delete;
DROP TRIGGER on_profile_delete;
DROP TRIGGER on_project_delete;
DROP TRIGGER on_storage_bucket_delete;
DROP TRIGGER on_storage_pool_delete;
DROP TRIGGER on_storage_volume_backup_delete;
DROP TRIGGER on_storage_volume_delete;
DROP TRIGGER on_storage_volume_snapshot_delete;
DROP TRIGGER on_warning_delete;
DROP TABLE identities_projects;
DROP TABLE auth_groups_permissions;
DROP TABLE auth_groups_identity_provider_groups;
DROP TABLE identities_auth_groups;
DROP TABLE identity_provider_groups;
DROP TABLE identities;
DROP TABLE auth_groups;
Rewrite commands:
Migration started
Stopping the source server
Stopping the target server
Unmounting "/var/lib/incus/devlxd"
Unmounting "/var/lib/incus/shmounts"
Wiping the target server
Migrating the data
Moving data over
Migrating database files
Writing the database patch
Cleaning up target paths
Cleaning up path "/var/lib/incus/backups"
Cleaning up path "/var/lib/incus/images"
Cleaning up path "/var/lib/incus/devices"
Cleaning up path "/var/lib/incus/devlxd"
Cleaning up path "/var/lib/incus/security"
Cleaning up path "/var/lib/incus/shmounts"
Rewrite symlinks:
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
Rewrite symlinks:
 - "/var/lib/incus/storage-pools/default/containers-snapshots/.." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/.." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/.." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/.." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/.." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/.." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/.." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/." to "/var/lib/incus/snapshots/."
Rewrite symlinks:
Rewrite symlinks:
Starting the target server
ERROR: Failed to run: systemctl start incus.service incus.socket: exit status 1 (Job for incus.service canceled.)
6

@stgraber I am hitting the exact same migration issue as @Heluca down to the strange volatile.uuid unsupported option error post migration when it tries to start the container. Is volatile.uuid not used by incus? Can incus ignore and drop this value for compat?

I am migrating from 5.21.2 lxd. Thanks.

7

volatile.uuid isn’t something that Incus uses indeed.

We do have a migration step that clears it from all volumes and volume snapshots though.
What do you see it still be set on?

8

@stgraber Pretty sure the volatile.uuid was thrown for the container in the same path as Heluca.

1 Like
9

lxd-to-incus: Clear volatile.uuid from instances by stgraber · Pull Request #1546 · lxc/incus · GitHub will clear from instances too.

1 Like