Today out of the blue, I found that my LXD isn’t working anymore. It continuously restarts with the this log:
time="2024-05-11T17:29:12+02:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-05-11T17:29:14+02:00" level=error msg="Failed initializing network" err="Failed starting: Failed loading instance config: instance_id" network=lxdbr0 project=default
time="2024-05-11T17:29:14+02:00" level=error msg="Failed to start the daemon" err="Failed loading local instances: Failed loading instance config: instance_id"
time="2024-05-11T17:29:14+02:00" level=warning msg="Loading local instances from disk as database is not available" err="Failed loading instance config: instance_id"
# lxd --debug --group lxd
DEBUG [2024-05-11T17:35:07+02:00] Connecting to a local LXD over a Unix socket
DEBUG [2024-05-11T17:35:07+02:00] Sending request to LXD etag= method=GET url="http://unix.socket/1.0"
INFO [2024-05-11T17:35:07+02:00] LXD is starting mode=normal path=/var/snap/lxd/common/lxd version=5.21.1
INFO [2024-05-11T17:35:07+02:00] Kernel uid/gid map:
INFO [2024-05-11T17:35:07+02:00] - u 0 0 4294967295
INFO [2024-05-11T17:35:07+02:00] - g 0 0 4294967295
INFO [2024-05-11T17:35:07+02:00] Configured LXD uid/gid map:
INFO [2024-05-11T17:35:07+02:00] - u 0 1000000 1000000000
INFO [2024-05-11T17:35:07+02:00] - g 0 1000000 1000000000
INFO [2024-05-11T17:35:07+02:00] Kernel features:
INFO [2024-05-11T17:35:07+02:00] - closing multiple file descriptors efficiently: yes
INFO [2024-05-11T17:35:07+02:00] - netnsid-based network retrieval: yes
INFO [2024-05-11T17:35:07+02:00] - pidfds: yes
INFO [2024-05-11T17:35:07+02:00] - core scheduling: yes
INFO [2024-05-11T17:35:07+02:00] - uevent injection: yes
INFO [2024-05-11T17:35:07+02:00] - seccomp listener: yes
INFO [2024-05-11T17:35:07+02:00] - seccomp listener continue syscalls: yes
INFO [2024-05-11T17:35:07+02:00] - seccomp listener add file descriptors: yes
INFO [2024-05-11T17:35:07+02:00] - attach to namespaces via pidfds: yes
INFO [2024-05-11T17:35:07+02:00] - safe native terminal allocation : yes
INFO [2024-05-11T17:35:07+02:00] - unprivileged file capabilities: yes
INFO [2024-05-11T17:35:07+02:00] - cgroup layout: cgroup2
WARNING[2024-05-11T17:35:07+02:00] - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored
WARNING[2024-05-11T17:35:07+02:00] - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead
INFO [2024-05-11T17:35:07+02:00] - idmapped mounts kernel support: yes
INFO [2024-05-11T17:35:07+02:00] Instance type operational driver=lxc features="map[]" type=container
ERROR [2024-05-11T17:35:08+02:00] Unable to run feature checks during QEMU initialization: Unable to locate the file for firmware "OVMF_CODE.4MB.fd"
WARNING[2024-05-11T17:35:08+02:00] Instance type not operational driver=qemu err="QEMU failed to run feature checks" type=virtual-machine
INFO [2024-05-11T17:35:08+02:00] Initializing local database
DEBUG [2024-05-11T17:35:08+02:00] Refreshing identity cache with local trusted certificates
INFO [2024-05-11T17:35:08+02:00] Set client certificate to server certificate fingerprint=f8e5967a4e24816664bcf6cbd25fea069c4b248cd38497967c4324bd10902424
DEBUG [2024-05-11T17:35:08+02:00] Initializing database gateway
INFO [2024-05-11T17:35:08+02:00] Starting database node id=1 local=1 role=voter
INFO [2024-05-11T17:35:08+02:00] Loading daemon configuration
DEBUG [2024-05-11T17:35:08+02:00] Connecting to a local LXD over a Unix socket
DEBUG [2024-05-11T17:35:08+02:00] Sending request to LXD etag= method=GET url="http://unix.socket/1.0"
DEBUG [2024-05-11T17:35:08+02:00] Detected stale unix socket, deleting
INFO [2024-05-11T17:35:08+02:00] Binding socket socket=/var/snap/lxd/common/lxd/unix.socket type="REST API Unix socket"
INFO [2024-05-11T17:35:08+02:00] Binding socket socket=/var/snap/lxd/common/lxd/devlxd/sock type="devlxd socket"
INFO [2024-05-11T17:35:08+02:00] Initializing global database
INFO [2024-05-11T17:35:08+02:00] Connecting to global database
DEBUG [2024-05-11T17:35:08+02:00] Dqlite: attempt 1: server 1: connected
INFO [2024-05-11T17:35:08+02:00] Connected to global database
INFO [2024-05-11T17:35:08+02:00] Initialized global database
DEBUG [2024-05-11T17:35:08+02:00] Firewall detected "xtables" incompatibility: Backend command "iptables" is an nftables shim
INFO [2024-05-11T17:35:08+02:00] Firewall loaded driver driver=nftables
INFO [2024-05-11T17:35:08+02:00] Initializing storage pools
DEBUG [2024-05-11T17:35:08+02:00] Initializing storage pool pool=zfs
DEBUG [2024-05-11T17:35:08+02:00] Mount started driver=zfs pool=zfs
DEBUG [2024-05-11T17:35:09+02:00] Mount finished driver=zfs pool=zfs
INFO [2024-05-11T17:35:09+02:00] Initialized storage pool pool=zfs
INFO [2024-05-11T17:35:09+02:00] All storage pools initialized
INFO [2024-05-11T17:35:09+02:00] Initializing daemon storage mounts
INFO [2024-05-11T17:35:09+02:00] Initializing networks
DEBUG [2024-05-11T17:35:09+02:00] Start driver=bridge network=lxdbr0 project=default
DEBUG [2024-05-11T17:35:09+02:00] Setting up network driver=bridge network=lxdbr0 project=default
DEBUG [2024-05-11T17:35:09+02:00] Stable MAC generated driver=bridge hwAddr="00:16:3e:b8:58:56" network=lxdbr0 project=default seed=f8e5967a4e24816664bcf6cbd25fea069c4b248cd38497967c4324bd10902424.0.1
DEBUG [2024-05-11T17:35:09+02:00] Clearing firewall driver=bridge network=lxdbr0 project=default
DEBUG [2024-05-11T17:35:09+02:00] Database error err="Failed loading instance config: instance_id"
ERROR [2024-05-11T17:35:09+02:00] Failed initializing network err="Failed starting: Failed loading instance config: instance_id" network=lxdbr0 project=default
DEBUG [2024-05-11T17:35:09+02:00] New operation class=task description="Pruning leftover image files" operation=6b129ab7-0d58-418d-9704-925ca9942873 project=
DEBUG [2024-05-11T17:35:09+02:00] Acquiring image task lock
DEBUG [2024-05-11T17:35:09+02:00] Acquired image task lock
INFO [2024-05-11T17:35:09+02:00] Cleaning up leftover image files
DEBUG [2024-05-11T17:35:09+02:00] Started operation class=task description="Pruning leftover image files" operation=6b129ab7-0d58-418d-9704-925ca9942873 project=
DEBUG [2024-05-11T17:35:09+02:00] Success for operation class=task description="Pruning leftover image files" operation=6b129ab7-0d58-418d-9704-925ca9942873 project=
INFO [2024-05-11T17:35:09+02:00] Done cleaning up leftover image files
INFO [2024-05-11T17:35:09+02:00] Starting device monitor
INFO [2024-05-11T17:35:09+02:00] Initialized filesystem monitor driver=fanotify path=/dev
DEBUG [2024-05-11T17:35:09+02:00] Database error err="Failed loading instance config: instance_id"
ERROR [2024-05-11T17:35:09+02:00] Failed to start the daemon err="Failed loading local instances: Failed loading instance config: instance_id"
INFO [2024-05-11T17:35:09+02:00] Starting shutdown sequence signal=interrupt
DEBUG [2024-05-11T17:35:09+02:00] Database error err="Failed loading instance config: instance_id"
WARNING[2024-05-11T17:35:09+02:00] Loading local instances from disk as database is not available err="Failed loading instance config: instance_id"
DEBUG [2024-05-11T17:35:09+02:00] Cancel ongoing or future gRPC connection attempts
DEBUG [2024-05-11T17:35:09+02:00] Cancel ongoing or future gRPC connection attempts
INFO [2024-05-11T17:35:09+02:00] Closing the database
INFO [2024-05-11T17:35:09+02:00] Stop database gateway
INFO [2024-05-11T17:35:09+02:00] Closing socket socket=/var/snap/lxd/common/lxd/unix.socket type="REST API Unix socket"
INFO [2024-05-11T17:35:09+02:00] Closing socket socket=/var/snap/lxd/common/lxd/devlxd/sock type="devlxd socket"
INFO [2024-05-11T17:35:09+02:00] Unmounting temporary filesystems
INFO [2024-05-11T17:35:09+02:00] Done unmounting temporary filesystems
INFO [2024-05-11T17:35:09+02:00] Daemon stopped
Error: Failed loading local instances: Failed loading instance config: instance_id
I tried to do the update from Ubuntu 22.04 LTS to Ubuntu 24,04 LTS (which I am now using) in the hope that it would go away, but it did not.
I further tried to restore “/var/snap/lxd/common/lxd/database/global/” from an older snapshot to no avail.
I am unsure how to proceed from here, I fail to find anything on the “Failed loading instance config”.
I do have some ZFS snapshots from both the containers and the snap directory.