Lxd-to-incus migration failed: Failed to initialize global database

Heluca · December 16, 2024, 12:42am

Hi, I migrated another system from from LXD to Incus and it ran perfectly, but today I tried migrating and noted that my version of LXD was too new. I ran “snap refresh lxd --channel=5.21/stable” to downgrade and everything seemed ok, so ran lxd-to-incus which appeared to complete migrations, but failed to start incus.

The error in the log file (/var/log/incus/incusd.log) is:
level=error msg=“Failed to start the daemon” err=“Failed to initialize global database: Failed to prepare statements: "\nDELETE FROM instances WHERE project_id = (SELECT projects.id FROM projects WHERE projects.name = ?) AND name = ?\n": no such table: main.auth_groups_permissions”

stgraber · December 16, 2024, 3:00am

Can you show the lxd-to-incus log, it should be in /var/log/.

We can’t look at the LXD code past 5.21 without risking legal complications which is why 5.21 is the last LXD release we support upgrading from. If the LXD team made any database change in releases after that, then the downgrade is unsafe and the database schema may be quite messed up.

Heluca · December 16, 2024, 4:26am

Thanks for the quick reply. It took a few hours, but I’m up and running again. I moved the database files and Incus started creating a new blank database. I used the incus admin recover to restore my containers from storage, but I had to manually edit each backup.yaml to remove the lines that reference “volatile.uuid” before it would work. I had 1 container that refused to start with a container snapshot miscount, and figured out I just needed to delete the reference. I reattached the network bridge to each container and networking worked normally. I still need to recreate a profile, but that can wait.
I connected to a remote Incus instance and copied containers without issue and my apps all seem to be running normally, so it looks healthy.
I only wish I hadn’t put off migrating. Thanks again.

Heluca · December 16, 2024, 4:34am

Here is the log. Everything seemed to be going well:

Source server: snap package
Target server: systemd
Source server paths: &{daemon:/var/snap/lxd/common/lxd logs:/var/snap/lxd/common/lxd/logs cache:/var/snap/lxd/common/lxd/cache}
Target server paths: &{daemon:/var/lib/incus logs:/var/log/incus cache:/var/cache/incus}
Rewrite SQL statements:
 - UPDATE profiles SET description='Default Incus profile' WHERE description='Default LXD profile';
 - UPDATE projects SET description='Default Incus project' WHERE description='Default LXD project';
 - DELETE FROM storage_volumes_config WHERE key='volatile.uuid';
 - DELETE FROM storage_volumes_snapshots_config WHERE key='volatile.uuid';
 - CREATE TABLE certificates (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    fingerprint TEXT NOT NULL,
    type INTEGER NOT NULL,
    name TEXT NOT NULL,
    certificate TEXT NOT NULL,
    restricted INTEGER NOT NULL DEFAULT 0,
    UNIQUE (fingerprint)
);
CREATE TABLE "certificates_projects" (
    certificate_id INTEGER NOT NULL,
    project_id INTEGER NOT NULL,
    FOREIGN KEY (certificate_id) REFERENCES certificates (id) ON DELETE CASCADE,
    FOREIGN KEY (project_id) REFERENCES "projects" (id) ON DELETE CASCADE,
    UNIQUE (certificate_id, project_id)
);
 - DELETE FROM schema WHERE version < 73;
UPDATE schema SET version=69 WHERE version=73;
 - INSERT INTO certificates (id, fingerprint, type, name, certificate, restricted) SELECT id, identifier, 1, name, json_extract(metadata, "$.cert"), 1 FROM identities WHERE type=1;
INSERT INTO certificates (id, fingerprint, type, name, certificate, restricted) SELECT id, identifier, 1, name, json_extract(metadata, "$.cert"), 0 FROM identities WHERE type=2;
INSERT INTO certificates (id, fingerprint, type, name, certificate, restricted) SELECT id, identifier, 2, name, json_extract(metadata, "$.cert"), 0 FROM identities WHERE type=3;
INSERT INTO certificates (id, fingerprint, type, name, certificate, restricted) SELECT id, identifier, 3, name, json_extract(metadata, "$.cert"), 1 FROM identities WHERE type=4;
INSERT INTO certificates (id, fingerprint, type, name, certificate, restricted) SELECT id, identifier, 3, name, json_extract(metadata, "$.cert"), 0 FROM identities WHERE type=6;
INSERT INTO certificates_projects (certificate_id, project_id) SELECT identity_id, project_id FROM identities_projects;
 - DROP TRIGGER on_auth_group_delete;
DROP TRIGGER on_cluster_group_delete;
DROP TRIGGER on_identity_delete;
DROP TRIGGER on_identity_provider_group_delete;
DROP TRIGGER on_image_alias_delete;
DROP TRIGGER on_image_delete;
DROP TRIGGER on_instance_backup_delete;
DROP TRIGGER on_instance_delete;
DROP TRIGGER on_instance_snaphot_delete;
DROP TRIGGER on_network_acl_delete;
DROP TRIGGER on_network_delete;
DROP TRIGGER on_network_zone_delete;
DROP TRIGGER on_node_delete;
DROP TRIGGER on_operation_delete;
DROP TRIGGER on_profile_delete;
DROP TRIGGER on_project_delete;
DROP TRIGGER on_storage_bucket_delete;
DROP TRIGGER on_storage_pool_delete;
DROP TRIGGER on_storage_volume_backup_delete;
DROP TRIGGER on_storage_volume_delete;
DROP TRIGGER on_storage_volume_snapshot_delete;
DROP TRIGGER on_warning_delete;
DROP TABLE identities_projects;
DROP TABLE auth_groups_permissions;
DROP TABLE auth_groups_identity_provider_groups;
DROP TABLE identities_auth_groups;
DROP TABLE identity_provider_groups;
DROP TABLE identities;
DROP TABLE auth_groups;
Rewrite commands:
Migration started
Stopping the source server
Stopping the target server
Unmounting "/var/lib/incus/devlxd"
Unmounting "/var/lib/incus/shmounts"
Wiping the target server
Migrating the data
Moving data over
Migrating database files
Writing the database patch
Cleaning up target paths
Cleaning up path "/var/lib/incus/backups"
Cleaning up path "/var/lib/incus/images"
Cleaning up path "/var/lib/incus/devices"
Cleaning up path "/var/lib/incus/devlxd"
Cleaning up path "/var/lib/incus/security"
Cleaning up path "/var/lib/incus/shmounts"
Rewrite symlinks:
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
 - "/var/lib/incus/storage-pools/default/containers/..." to "/var/lib/incus/containers/.."
Rewrite symlinks:
 - "/var/lib/incus/storage-pools/default/containers-snapshots/.." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/.." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/.." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/.." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/.." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/.." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/.." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/." to "/var/lib/incus/snapshots/."
 - "/var/lib/incus/storage-pools/default/containers-snapshots/." to "/var/lib/incus/snapshots/."
Rewrite symlinks:
Rewrite symlinks:
Starting the target server
ERROR: Failed to run: systemctl start incus.service incus.socket: exit status 1 (Job for incus.service canceled.)

Diegmontoya · December 22, 2024, 8:04pm

@stgraber I am hitting the exact same migration issue as @Heluca down to the strange volatile.uuid unsupported option error post migration when it tries to start the container. Is volatile.uuid not used by incus? Can incus ignore and drop this value for compat?

I am migrating from 5.21.2 lxd. Thanks.

stgraber · December 23, 2024, 11:53am

volatile.uuid isn’t something that Incus uses indeed.

We do have a migration step that clears it from all volumes and volume snapshots though.
What do you see it still be set on?

Diegmontoya · December 24, 2024, 10:34am

@stgraber Pretty sure the volatile.uuid was thrown for the container in the same path as Heluca.

stgraber · December 24, 2024, 12:19pm

lxd-to-incus: Clear volatile.uuid from instances by stgraber · Pull Request #1546 · lxc/incus · GitHub will clear from instances too.

Lox · March 20, 2025, 8:23am

Just had the issue. lxd-to-incus wiped the whole config

incus admin recover failed on volatile.uuid … Now giving lxd recover a try.

stgraber · March 20, 2025, 1:43pm

Best course of action if you get a lxd-to-incus failure is to not try to do anything rushed at that stage, but report the error here and provide the log that’s written to /var/log/.

The tool takes all the backups needed to do a full revert, but in most cases, it’s easier to fix/workaround issues to get the migration to complete than try to revert it.

disputin · April 10, 2025, 4:53pm

I don’t want to steal a thread, but I’m having the same issue. Here is my lxc-to-incus.log

Source server: snap package
Target server: systemd
Source server paths: &{daemon:/var/snap/lxd/common/lxd logs:/var/snap/lxd/common/lxd/logs cache:/var/snap/lxd/common/lxd/cache}
Target server paths: &{daemon:/var/lib/incus logs:/var/log/incus cache:/var/cache/incus}
Rewrite SQL statements:
 - UPDATE profiles SET description='Default Incus profile' WHERE description='Default LXD profile';
 - UPDATE projects SET description='Default Incus project' WHERE description='Default LXD project';
 - DELETE FROM storage_volumes_config WHERE key='volatile.uuid';
 - DELETE FROM storage_volumes_snapshots_config WHERE key='volatile.uuid';
 - CREATE TABLE certificates (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    fingerprint TEXT NOT NULL,
    type INTEGER NOT NULL,
    name TEXT NOT NULL,
    certificate TEXT NOT NULL,
    restricted INTEGER NOT NULL DEFAULT 0,
    UNIQUE (fingerprint)
);
CREATE TABLE "certificates_projects" (
    certificate_id INTEGER NOT NULL,
    project_id INTEGER NOT NULL,
    FOREIGN KEY (certificate_id) REFERENCES certificates (id) ON DELETE CASCADE,
    FOREIGN KEY (project_id) REFERENCES "projects" (id) ON DELETE CASCADE,
    UNIQUE (certificate_id, project_id)
);
 - DELETE FROM schema WHERE version < 73;
UPDATE schema SET version=69 WHERE version=73;
 - INSERT INTO certificates (id, fingerprint, type, name, certificate, restricted) SELECT id, identifier, 1, name, json_extract(metadata, "$.cert"
), 1 FROM identities WHERE type=1;
INSERT INTO certificates (id, fingerprint, type, name, certificate, restricted) SELECT id, identifier, 1, name, json_extract(metadata, "$.cert"), 
0 FROM identities WHERE type=2;
INSERT INTO certificates (id, fingerprint, type, name, certificate, restricted) SELECT id, identifier, 2, name, json_extract(metadata, "$.cert"), 
0 FROM identities WHERE type=3;
INSERT INTO certificates (id, fingerprint, type, name, certificate, restricted) SELECT id, identifier, 3, name, json_extract(metadata, "$.cert"), 
1 FROM identities WHERE type=4;
INSERT INTO certificates (id, fingerprint, type, name, certificate, restricted) SELECT id, identifier, 3, name, json_extract(metadata, "$.cert"), 
0 FROM identities WHERE type=6;
INSERT INTO certificates_projects (certificate_id, project_id) SELECT identity_id, project_id FROM identities_projects;
 - DROP TRIGGER on_auth_group_delete;
DROP TRIGGER on_cluster_group_delete;
DROP TRIGGER on_identity_delete;
DROP TRIGGER on_identity_provider_group_delete;
DROP TRIGGER on_image_alias_delete;
DROP TRIGGER on_image_delete;
DROP TRIGGER on_instance_backup_delete;
DROP TRIGGER on_instance_delete;
DROP TRIGGER on_instance_snaphot_delete;
DROP TRIGGER on_network_acl_delete;
DROP TRIGGER on_network_delete;
DROP TRIGGER on_network_zone_delete;
DROP TRIGGER on_node_delete;
DROP TRIGGER on_operation_delete;
DROP TRIGGER on_profile_delete;
DROP TRIGGER on_project_delete;
DROP TRIGGER on_storage_bucket_delete;
DROP TRIGGER on_storage_pool_delete;
DROP TRIGGER on_storage_volume_backup_delete;
DROP TRIGGER on_storage_volume_delete;
DROP TRIGGER on_storage_volume_snapshot_delete;
DROP TRIGGER on_warning_delete;
DROP TABLE identities_projects;
DROP TABLE auth_groups_permissions;
DROP TABLE auth_groups_identity_provider_groups;
DROP TABLE identities_auth_groups;
DROP TABLE identity_provider_groups;
DROP TABLE identities;
DROP TABLE auth_groups;
Rewrite commands:
Migration started
Stopping the source server
Stopping the target server
Unmounting "/var/lib/incus/devlxd"
Unmounting "/var/lib/incus/shmounts"
Wiping the target server
Migrating the data
Moving data over
Migrating database files
Writing the database patch
Cleaning up target paths
Cleaning up path "/var/lib/incus/backups"
Cleaning up path "/var/lib/incus/images"
Cleaning up path "/var/lib/incus/devices"
Cleaning up path "/var/lib/incus/devlxd"
Cleaning up path "/var/lib/incus/security"
Cleaning up path "/var/lib/incus/shmounts"
Rewrite symlinks:
 - "/var/lib/incus/storage-pools/lxd-storage/containers/acs" to "/var/lib/incus/containers/acs"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/ansible" to "/var/lib/incus/containers/ansible"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/ansible-ll1" to "/var/lib/incus/containers/ansible-ll1"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/ansible-ll2" to "/var/lib/incus/containers/ansible-ll2"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/ansible-ll3" to "/var/lib/incus/containers/ansible-ll3"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/ansible-test-bi" to "/var/lib/incus/containers/ansible-test-bi"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/ansible-test1" to "/var/lib/incus/containers/ansible-test1"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/ansible-test2" to "/var/lib/incus/containers/ansible-test2"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/ansible-test3" to "/var/lib/incus/containers/ansible-test3"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/ansible-test4" to "/var/lib/incus/containers/ansible-test4"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/ansible-topology" to "/var/lib/incus/containers/ansible-topology"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/baseimage" to "/var/lib/incus/containers/baseimage"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/basetopology" to "/var/lib/incus/containers/basetopology"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/cdcserv" to "/var/lib/incus/containers/cdcserv"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/connserv" to "/var/lib/incus/containers/connserv"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/dataagg" to "/var/lib/incus/containers/dataagg"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/datacol" to "/var/lib/incus/containers/datacol"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/docker-host" to "/var/lib/incus/containers/docker-host"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/gas" to "/var/lib/incus/containers/gas"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/gns3" to "/var/lib/incus/containers/gns3"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/hazelcast1" to "/var/lib/incus/containers/hazelcast1"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/hazelcast2" to "/var/lib/incus/containers/hazelcast2"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/kafka1" to "/var/lib/incus/containers/kafka1"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/microk8s" to "/var/lib/incus/containers/microk8s"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/neo4j" to "/var/lib/incus/containers/neo4j"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/portal" to "/var/lib/incus/containers/portal"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/spectrum" to "/var/lib/incus/containers/spectrum"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/terraform" to "/var/lib/incus/containers/terraform"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/topology" to "/var/lib/incus/containers/topology"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/vna" to "/var/lib/incus/containers/vna"
 - "/var/lib/incus/storage-pools/lxd-storage/containers/vpca" to "/var/lib/incus/containers/vpca"
Rewrite symlinks:
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/acs" to "/var/lib/incus/snapshots/acs"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/ansible" to "/var/lib/incus/snapshots/ansible"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/ansible-ll1" to "/var/lib/incus/snapshots/ansible-ll1"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/ansible-ll2" to "/var/lib/incus/snapshots/ansible-ll2"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/ansible-ll3" to "/var/lib/incus/snapshots/ansible-ll3"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/ansible-test-bi" to "/var/lib/incus/snapshots/ansible-test-bi"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/ansible-test1" to "/var/lib/incus/snapshots/ansible-test1"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/ansible-test2" to "/var/lib/incus/snapshots/ansible-test2"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/ansible-test3" to "/var/lib/incus/snapshots/ansible-test3"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/ansible-test4" to "/var/lib/incus/snapshots/ansible-test4"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/ansible-topology" to "/var/lib/incus/snapshots/ansible-topology"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/baseimage" to "/var/lib/incus/snapshots/baseimage"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/basetopology" to "/var/lib/incus/snapshots/basetopology"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/cdcserv" to "/var/lib/incus/snapshots/cdcserv"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/connserv" to "/var/lib/incus/snapshots/connserv"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/dataagg" to "/var/lib/incus/snapshots/dataagg"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/datacol" to "/var/lib/incus/snapshots/datacol"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/docker-host" to "/var/lib/incus/snapshots/docker-host"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/gas" to "/var/lib/incus/snapshots/gas"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/gns3" to "/var/lib/incus/snapshots/gns3"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/hazelcast1" to "/var/lib/incus/snapshots/hazelcast1"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/hazelcast2" to "/var/lib/incus/snapshots/hazelcast2"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/kafka1" to "/var/lib/incus/snapshots/kafka1"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/neo4j" to "/var/lib/incus/snapshots/neo4j"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/portal" to "/var/lib/incus/snapshots/portal"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/spectrum" to "/var/lib/incus/snapshots/spectrum"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/terraform" to "/var/lib/incus/snapshots/terraform"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/topology" to "/var/lib/incus/snapshots/topology"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/vna" to "/var/lib/incus/snapshots/vna"
 - "/var/lib/incus/storage-pools/lxd-storage/containers-snapshots/vpca" to "/var/lib/incus/snapshots/vpca"
Rewrite symlinks:
 - "/var/lib/incus/storage-pools/lxd-storage/virtual-machines/datarep" to "/var/lib/incus/virtual-machines/datarep"
Rewrite symlinks:
 - "/var/lib/incus/storage-pools/lxd-storage/virtual-machines-snapshots/datarep" to "/var/lib/incus/virtual-machines-snapshots/datarep"
Starting the target server
ERROR: Failed to run: systemctl start incus.service incus.socket: exit status 1 (Job for incus.service failed because the control process exited w
ith error code.
See "systemctl status incus.service" and "journalctl -xeu incus.service" for details.)

systemctl status incus shows:

● incus.service - Incus - Daemon
     Loaded: loaded (/usr/lib/systemd/system/incus.service; indirect; preset: disabled)
     Active: activating (start-post) (Result: exit-code) since Thu 2025-04-10 09:48:33 PDT; 4min 8s ago
TriggeredBy: ● incus.socket
       Docs: man:incusd(1)
    Process: 11377 ExecStart=/usr/libexec/incus/incusd --group incus-admin (code=exited, status=1/FAILURE)
   Main PID: 11377 (code=exited, status=1/FAILURE); Control PID: 11378 (incusd)
      Tasks: 7
     Memory: 24.6M
        CPU: 712ms
     CGroup: /system.slice/incus.service
             └─.control
               └─11378 /usr/libexec/incus/incusd waitready --timeout=600

Apr 10 09:48:33 dxnetops.seansdomain.org systemd[1]: Starting Incus - Daemon...
Apr 10 09:48:33 dxnetops.seansdomain.org incusd[11377]: time="2025-04-10T09:48:33-07:00" level=error msg="Unable to parse system idmap" err="No m>
Apr 10 09:48:33 dxnetops.seansdomain.org incusd[11377]: time="2025-04-10T09:48:33-07:00" level=warning msg="AppArmor support has been disabled be>
Apr 10 09:48:33 dxnetops.seansdomain.org incusd[11377]: time="2025-04-10T09:48:33-07:00" level=warning msg=" - AppArmor support has been disabled>
Apr 10 09:48:33 dxnetops.seansdomain.org incusd[11377]: time="2025-04-10T09:48:33-07:00" level=warning msg="Instance type not operational" driver>
Apr 10 09:48:34 dxnetops.seansdomain.org incusd[11377]: time="2025-04-10T09:48:34-07:00" level=error msg="Failed to start the daemon" err="Failed>
Apr 10 09:48:34 dxnetops.seansdomain.org incusd[11377]: Error: Failed to initialize global database: failed to ensure schema: failed to execute q>
Apr 10 09:48:34 dxnetops.seansdomain.org systemd[1]: incus.service: Main process exited, code=exited, status=1/FAILURE

stgraber · April 10, 2025, 5:07pm

Your systemctl output cuts off the part that actually says what the problem is…

Can you show journalctl -u incus -n 30?

disputin · April 10, 2025, 5:21pm

Apr 10 10:07:13 dxnetops.seansdomain.org incusd[24168]: time="2025-04-10T10:07:13-07:00" level=warning msg="Instance type not operational" driver=qemu err="QEMU command not available: exec: \"qemu-system-x86_64\": executable file not found in $PATH" type=virtual->
Apr 10 10:07:14 dxnetops.seansdomain.org incusd[24168]: time="2025-04-10T10:07:14-07:00" level=error msg="Failed to start the daemon" err="Failed to initialize global database: failed to ensure schema: failed to execute queries from /var/lib/incus/database/patch.>
Apr 10 10:07:14 dxnetops.seansdomain.org incusd[24168]: Error: Failed to initialize global database: failed to ensure schema: failed to execute queries from /var/lib/incus/database/patch.global.sql: no such trigger: on_instance_snaphot_delete
Apr 10 10:07:14 dxnetops.seansdomain.org systemd[1]: incus.service: Main process exited, code=exited, status=1/FAILURE
Apr 10 10:09:05 dxnetops.seansdomain.org systemd[1]: incus.service: Control process exited, code=killed, status=15/TERM
Apr 10 10:09:05 dxnetops.seansdomain.org systemd[1]: incus.service: Failed with result 'exit-code'.
Apr 10 10:09:05 dxnetops.seansdomain.org systemd[1]: Stopped Incus - Daemon.
Apr 10 10:09:05 dxnetops.seansdomain.org systemd[1]: Starting Incus - Daemon...
Apr 10 10:09:05 dxnetops.seansdomain.org incusd[25291]: time="2025-04-10T10:09:05-07:00" level=warning msg="AppArmor support has been disabled because of lack of kernel support"
Apr 10 10:09:05 dxnetops.seansdomain.org incusd[25291]: time="2025-04-10T10:09:05-07:00" level=warning msg=" - AppArmor support has been disabled, Disabled because of lack of kernel support"
Apr 10 10:09:05 dxnetops.seansdomain.org incusd[25291]: time="2025-04-10T10:09:05-07:00" level=error msg="Unable to run feature checks during QEMU initialization: Unable to locate a UEFI firmware"
Apr 10 10:09:05 dxnetops.seansdomain.org incusd[25291]: time="2025-04-10T10:09:05-07:00" level=warning msg="Instance type not operational" driver=qemu err="QEMU failed to run feature checks" type=virtual-machine
Apr 10 10:09:06 dxnetops.seansdomain.org incusd[25291]: time="2025-04-10T10:09:06-07:00" level=error msg="Failed to start the daemon" err="Failed to initialize global database: failed to ensure schema: failed to execute queries from /var/lib/incus/database/patch.>
Apr 10 10:09:06 dxnetops.seansdomain.org incusd[25291]: Error: Failed to initialize global database: failed to ensure schema: failed to execute queries from /var/lib/incus/database/patch.global.sql: no such trigger: on_instance_snaphot_delete
Apr 10 10:09:06 dxnetops.seansdomain.org systemd[1]: incus.service: Main process exited, code=exited, status=1/FAILURE
Apr 10 10:19:05 dxnetops.seansdomain.org incusd[25292]: Error: Daemon still not running after 600s timeout (Get "http://unix.socket/1.0": EOF)
Apr 10 10:19:05 dxnetops.seansdomain.org systemd[1]: incus.service: start-post operation timed out. Terminating.
Apr 10 10:19:05 dxnetops.seansdomain.org systemd[1]: incus.service: Control process exited, code=exited, status=1/FAILURE
Apr 10 10:19:05 dxnetops.seansdomain.org systemd[1]: incus.service: Failed with result 'exit-code'.
Apr 10 10:19:05 dxnetops.seansdomain.org systemd[1]: Failed to start Incus - Daemon.
Apr 10 10:19:06 dxnetops.seansdomain.org systemd[1]: incus.service: Scheduled restart job, restart counter is at 1.
Apr 10 10:19:06 dxnetops.seansdomain.org systemd[1]: Stopped Incus - Daemon.
Apr 10 10:19:06 dxnetops.seansdomain.org systemd[1]: Starting Incus - Daemon...
Apr 10 10:19:06 dxnetops.seansdomain.org incusd[31891]: time="2025-04-10T10:19:06-07:00" level=warning msg="AppArmor support has been disabled because of lack of kernel support"
Apr 10 10:19:06 dxnetops.seansdomain.org incusd[31891]: time="2025-04-10T10:19:06-07:00" level=warning msg=" - AppArmor support has been disabled, Disabled because of lack of kernel support"
Apr 10 10:19:06 dxnetops.seansdomain.org incusd[31891]: time="2025-04-10T10:19:06-07:00" level=error msg="Unable to run feature checks during QEMU initialization: Unable to locate a UEFI firmware"
Apr 10 10:19:06 dxnetops.seansdomain.org incusd[31891]: time="2025-04-10T10:19:06-07:00" level=warning msg="Instance type not operational" driver=qemu err="QEMU failed to run feature checks" type=virtual-machine
Apr 10 10:19:06 dxnetops.seansdomain.org incusd[31891]: time="2025-04-10T10:19:06-07:00" level=error msg="Failed to start the daemon" err="Failed to initialize global database: failed to ensure schema: failed to execute queries from /var/lib/incus/database/patch.>
Apr 10 10:19:06 dxnetops.seansdomain.org incusd[31891]: Error: Failed to initialize global database: failed to ensure schema: failed to execute queries from /var/lib/incus/database/patch.global.sql: no such trigger: on_instance_snaphot_delete
Apr 10 10:19:06 dxnetops.seansdomain.org systemd[1]: incus.service: Main process exited, code=exited, status=1/FAILURE

stgraber · April 10, 2025, 5:27pm

Ah yeah, so that’s a typo in lxd-to-incus which has been fixed already.

I believe we have that fix in 6.0.4, it will also be in 6.12 and it’s been included in the 6.11 Zabbly packages too.

The issue linked from the fix also includes a manual workaround for systems that got hit by this: lxd-to-incus 6.11 hangs · Issue #78 · zabbly/incus · GitHub

disputin · April 10, 2025, 5:35pm

One step forward. Now I’m seeing

Apr 10 10:28:47 dxnetops.seansdomain.org systemd[1]: incus.service: Control process exited, code=killed, status=15/TERM
Apr 10 10:28:47 dxnetops.seansdomain.org systemd[1]: incus.service: Failed with result 'exit-code'.
Apr 10 10:28:47 dxnetops.seansdomain.org systemd[1]: Stopped Incus - Daemon.
Apr 10 10:28:47 dxnetops.seansdomain.org systemd[1]: Starting Incus - Daemon...
Apr 10 10:28:47 dxnetops.seansdomain.org incusd[38249]: time="2025-04-10T10:28:47-07:00" level=warning msg="AppArmor support has been disabled because of lack of kernel support"
Apr 10 10:28:47 dxnetops.seansdomain.org incusd[38249]: time="2025-04-10T10:28:47-07:00" level=warning msg=" - AppArmor support has been disabled, Disabled because of lack of kernel support"
Apr 10 10:28:47 dxnetops.seansdomain.org incusd[38249]: time="2025-04-10T10:28:47-07:00" level=error msg="Unable to run feature checks during QEMU initialization: Unable to locate a UEFI firmware"
Apr 10 10:28:47 dxnetops.seansdomain.org incusd[38249]: time="2025-04-10T10:28:47-07:00" level=warning msg="Instance type not operational" driver=qemu err="QEMU failed to run feature checks" type=virtual-machine
Apr 10 10:28:48 dxnetops.seansdomain.org incusd[38249]: time="2025-04-10T10:28:48-07:00" level=error msg="Failed to start the daemon" err="Failed to initialize global database: Failed to prepare statements: \"\\nDELETE FROM instances_snapshots WHERE instance_id = (SELECT instances.id FROM instances JOIN projects ON instances.project_id = projects.id WHERE projects.name = ? AND instances.name = ?) AND name = ?\\n\": no such table: main.auth_groups_permissions"
Apr 10 10:28:48 dxnetops.seansdomain.org incusd[38249]: Error: Failed to initialize global database: Failed to prepare statements: "\nDELETE FROM instances_snapshots WHERE instance_id = (SELECT instances.id FROM instances JOIN projects ON instances.project_id = projects.id WHERE projects.name = ? AND instances.name = ?) AND name = ?\n": no such table: main.auth_groups_permissions
Apr 10 10:28:48 dxnetops.seansdomain.org systemd[1]: incus.service: Main process exited, code=exited, status=1/FAILURE```

stgraber · April 10, 2025, 8:03pm

incus admin sql global .dump | grep auth_