'lxd activateifneeded' fails with database errors, everything else works

I am a new user of LXD. While testing it, I encountered a strange issue; I am not sure whether I did something wrong, or bumped into a bug.

I am using Ubuntu 18.04.4, kernel 5.3.0-40-generic. At first I installed lxd package from Ubuntu repo (version 3.0.3). That worked with no problems, but did not support container export/import, so I tried a newer version. I downloaded lxd-3.21.tar.gz, compiled it following instructions at https://github.com/lxc/lxd/, gathered executables and libraries and manually installed them under /opt, created systemd unit files with modified paths (lxd.service and lxd-containers.service), and made sure that PATH and LD_LIBRARY_PATH were set correctly. I did not uninstall the 3.0.3 package and I did not compile a new version of lxcfs.

After that lxd.service works with no issues and ‘lxc version’ shows 3.21 for both client and server. But lxd-containers.service fails on startup every time, and so does ‘lxd activateifneeded’. This does not depend whether the lxd daemon is running or not, or whether any containers are running or not. One of the following error messages is displayed (or logged). Bizarrely, subsequent invocations of ‘lxd activateifneeded’ produced different errors (in the order as in the list below), but then stabilized on “Failed to fetch instances: sql: Scan error on column index 11”. I suspected a problem with DB schema, but according to the lxd log the schema was updated properly. Surprisingly, the same executable called with different parameters (‘lxd’, ‘lxd waitready’ and ‘lxd shutdown’) works fine and produces no errors.

To work around this issue I disabled lxd-containers.service and modified lxd.service to start on system boot, rather than rely on socket activation. With such a config everything works fine.

I will gladly supply additional information if needed. Thanks for help.

List of errors produced by ‘lxd activateifneeded’ (in order of appearance):

Error: Prepare database statements: “\nSELECT projects.description, projects.name\n FROM projects\n ORDER BY projects.name\n”: no such table: projects

Error: Prepare database statements: “\nSELECT project, name, device, type, key, value FROM instances_devices_ref ORDER BY project, name\n”: no such table: instances_devices_ref

Error: Prepare database statements: “\nSELECT projects.name AS project, profiles.name\n FROM profiles JOIN projects ON profiles.project_id = projects.id\n WHERE project = ? AND profiles.name = ? ORDER BY projects.id, profiles.name\n”: no such table: projects

Error: Prepare database statements: “\nSELECT instances_snapshots.id, projects.name AS project, instances.name AS instance, instances_snapshots.name, instances_snapshots.creation_date, instances_snapshots.stateful, coalesce(instances_snapshots.description, ‘’), instances_snapshots.expiry_date\n FROM instances_snapshots JOIN projects ON instances.project_id = projects.id JOIN instances ON instances_snapshots.instance_id = instances.id\n WHERE project = ? AND instance = ? AND instances_snapshots.name = ? ORDER BY projects.id, instances.id, instances_snapshots.name\n”: no such table: instances_snapshots

Error: Prepare database statements: “\nSELECT project, name, value FROM instances_profiles_ref WHERE project = ? ORDER BY project, name\n”: no such table: instances_profiles_ref

Error: Prepare database statements: “\nSELECT instances.id, projects.name AS project, instances.name, nodes.name AS node, instances.type, instances.architecture, instances.ephemeral, instances.creation_date, instances.stateful, instances.last_use_date, coalesce(instances.description, ‘’), instances.expiry_date\n FROM instances JOIN projects ON instances.project_id = projects.id JOIN nodes ON instances.node_id = nodes.id\n WHERE instances.type = ? AND instances.name = ? AND node = ? ORDER BY projects.id, instances.name\n”: no such table: instances

Error: Prepare database statements: “\nSELECT project, name, value FROM instances_profiles_ref ORDER BY project, name\n”: no such table: instances_profiles_ref

Error: Prepare database statements: “\nUPDATE instances SET name = ? WHERE project_id = (SELECT projects.id FROM projects WHERE projects.name = ?) AND name = ?\n”: no such table: instances

Error: Failed to fetch instances: sql: Scan error on column index 11: unsupported Scan, storing driver.Value type into type *time.Time

I cannot reproduce the problem on a clean Ubuntu 18.04.4 install with LXD 3.21 installed as as snap. I will try a clean install with LXD and LXCFS compiled from source.