Lxd-containers service fails to start after upgrade to Fedora 32

Some additional info: I just upgraded from Fedora 31 to 32 and I’m getting LXD packages from ganto-lxc3 and the default Fedora repos.

I had to downgrade lxcfs from 4.0.4 to 4.0.1 due to an issue with “undefined symbol: lxcfs_clone”. That issue seems to have been resolved back in June so I’m not sure why I’m still experiencing it (https://github.com/lxc/lxcfs/issues/420). The downgrade resolved that, however.

The main issue I’m having is now with the lxd-containers service. Service status:

● lxd-containers.service - LXD - container startup/shutdown
     Loaded: loaded (/usr/lib/systemd/system/lxd-containers.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Tue 2020-09-29 18:02:14 EDT; 53min ago
       Docs: man:lxd(1)
    Process: 6338 ExecStart=/usr/bin/lxd activateifneeded (code=exited, status=1/FAILURE)
   Main PID: 6338 (code=exited, status=1/FAILURE)

Sep 29 18:02:13 localhost.localdomain systemd[1]: Starting LXD - container startup/shutdown...
Sep 29 18:02:14 localhost.localdomain lxd[6338]: Error: Get "http://unix.socket/1.0": EOF
Sep 29 18:02:14 localhost.localdomain systemd[1]: lxd-containers.service: Main process exited, code=exited, status=1/FAILURE
Sep 29 18:02:14 localhost.localdomain systemd[1]: lxd-containers.service: Failed with result 'exit-code'.
Sep 29 18:02:14 localhost.localdomain systemd[1]: Failed to start LXD - container startup/shutdown.

And a section from lxd.log:

t=2020-09-29T18:52:14-0400 lvl=info msg="LXD 3.20 is starting in normal mode" path=/var/lib/lxd
t=2020-09-29T18:52:14-0400 lvl=info msg="Kernel uid/gid map:" 
t=2020-09-29T18:52:14-0400 lvl=info msg=" - u 0 0 4294967295" 
t=2020-09-29T18:52:14-0400 lvl=info msg=" - g 0 0 4294967295" 
t=2020-09-29T18:52:14-0400 lvl=info msg="Configured LXD uid/gid map:" 
t=2020-09-29T18:52:14-0400 lvl=info msg=" - u 0 1000000 65536" 
t=2020-09-29T18:52:14-0400 lvl=info msg=" - g 0 1000000 65536" 
t=2020-09-29T18:52:14-0400 lvl=warn msg="AppArmor support has been disabled because of lack of kernel support" 
t=2020-09-29T18:52:14-0400 lvl=info msg="Kernel features:" 
t=2020-09-29T18:52:14-0400 lvl=info msg=" - netnsid-based network retrieval: yes" 
t=2020-09-29T18:52:14-0400 lvl=info msg=" - uevent injection: yes" 
t=2020-09-29T18:52:14-0400 lvl=info msg=" - seccomp listener: yes" 
t=2020-09-29T18:52:14-0400 lvl=info msg=" - seccomp listener continue syscalls: yes" 
t=2020-09-29T18:52:14-0400 lvl=info msg=" - unprivileged file capabilities: yes" 
t=2020-09-29T18:52:14-0400 lvl=info msg=" - cgroup layout: hybrid" 
t=2020-09-29T18:52:14-0400 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, I/O weight limits will be ignored" 
t=2020-09-29T18:52:14-0400 lvl=info msg=" - shiftfs support: no" 
t=2020-09-29T18:52:14-0400 lvl=info msg="Initializing local database" 
t=2020-09-29T18:52:15-0400 lvl=info msg="Starting /dev/lxd handler:" 
t=2020-09-29T18:52:15-0400 lvl=info msg=" - binding devlxd socket" socket=/var/lib/lxd/devlxd/sock
t=2020-09-29T18:52:15-0400 lvl=info msg="REST API daemon:" 
t=2020-09-29T18:52:15-0400 lvl=info msg=" - binding Unix socket" inherited=true socket=/run/lxd.socket
t=2020-09-29T18:52:15-0400 lvl=info msg="Initializing global database" 
t=2020-09-29T18:52:15-0400 lvl=eror msg="Failed to start the daemon: failed to open cluster database: failed to ensure schema: schema version '26' is more recent than expected '23'" 
t=2020-09-29T18:52:15-0400 lvl=info msg="Starting shutdown sequence" 
t=2020-09-29T18:52:15-0400 lvl=info msg="Stopping REST API handler:" 
t=2020-09-29T18:52:15-0400 lvl=info msg=" - closing socket" socket=/run/lxd.socket
t=2020-09-29T18:52:15-0400 lvl=info msg="Stopping /dev/lxd handler:" 
t=2020-09-29T18:52:15-0400 lvl=info msg=" - closing socket" socket=/var/lib/lxd/devlxd/sock

Is the issue with the schema version? I didn’t see any topics related to that error.

The error suggested that LXD itself got downgraded too which isn’t something that’s supported due to database schemas evolving and an older version of LXD being unable to read the data from a more recent version.

I’m not sure how LXD would have been downgraded. I have version 3.20 installed, and that’s the latest listed in the ganto-lxc3 repo.

It seems like my issue might be related to this: https://github.com/ganto/copr-lxc3/issues/24

I’m not sure what the best path forward would be.

The ganto-lxc3 repo caused the downgrade. I updated to ganto-lx4 and everything is working again.