[SOLVED]Error: ZFS storage pool "default" could not be imported: cannot import 'default': no such pool available

Hi all,
Accidentally I disconnected the power of the machine and reboot again gives me an error as in the title. And cant start the lxd daemon, as I understand correctly storage.zfs_pool_name seems default, but my zpool name is lxdpool.

indiana@lxdserver01:~$ zpool status -v
pool: lxdpool
state: ONLINE
scan: none requested
config:

NAME        STATE     READ WRITE CKSUM
lxdpool     ONLINE       0     0     0
  sdb       ONLINE       0     0     0

errors: No known data errors

Here is my question, where can I find the configuration file of lxc/lxd of lxc info output and how can I overcome that error message and start lxd deamon without error.
Thanks.

Sounds like you’re using LXD 2.0.x, in which case, it’s stored in /var/lib/lxd/lxd.db as a sqlite3 database.

Something like sqlite3 /var/lib/lxd/lxd.db "UPDATE config SET value='lxdpool' WHERE key='storage.zfs_pool_name'" may do the trick.

@stgraber, Sorry not to mention. My lxd version is 3.0.1 therefore cant find the /var/lib/lxd/lxd.db file.
Where is the configuration file for 3.0.1?
Thanks.

indiana@lxdserver01:~$ lxd --version
3.0.1

Can you show the output of:

sqlite3 /var/lib/lxd/database/global/db.bin 'SELECT * FROM storage_pools_config;'

Note that you won’t be able to just update that database, so based on the output above I should be able to give you a small SQL patch to put in place on the filesystem which will then correct your database on next LXD startup.

Here is the output of the command. Thanks.

root@lxdserver01:/var/lib/lxd/database/global# sqlite3 /var/lib/lxd/database/global/db.bin ‘SELECT * FROM storage_pools_config;’
Error: no such table: storage_pools_config

Hmm, that’s odd, it does work fine here. I wonder if it’s because you’re halfway through an upgrade maybe.

Can you show:

  • find /var/lib/lxd/database/

Output of the system information, thanks.

root@lxdserver01:~# find /var/lib/lxd/database/
/var/lib/lxd/database/
/var/lib/lxd/database/local.db
/var/lib/lxd/database/global
/var/lib/lxd/database/global/logs.db
/var/lib/lxd/database/global/snapshots
/var/lib/lxd/database/global/snapshots/1-760-1536610021753
/var/lib/lxd/database/global/snapshots/1-760-1536610021753/meta.json
/var/lib/lxd/database/global/snapshots/1-760-1536610021753/state.bin
/var/lib/lxd/database/global/snapshots/1-762-1536610622008
/var/lib/lxd/database/global/snapshots/1-762-1536610622008/meta.json
/var/lib/lxd/database/global/snapshots/1-762-1536610622008/state.bin
/var/lib/lxd/database/global/db.bin

root@lxdserver01:~# more /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
root@lxdserver01:~# dpkg -l | grep -i lxd
ii  lxd                                   3.0.1-0ubuntu1~18.04.1            amd64        Container hypervisor based on LXC - daemon
ii  lxd-client                            3.0.1-0ubuntu1~18.04.1            amd64        Container hypervisor based on LXC - client

Can you show the output of (as root):

  • systemctl stop lxd.service lxd.socket
  • lxd --debug --group lxd

That should let me figure out why the database appears partly setup and then provide a better way to sort things out.

I just stopped and try to restart the service and socket, and here is the error log.

root@lxdserver01:~# lxd --debug --group lxd
DBUG[09-11|20:55:45] Connecting to a local LXD over a Unix socket
DBUG[09-11|20:55:45] Sending request to LXD etag= method=GET url=http://unix.socket/1.0
INFO[09-11|20:56:34] LXD 3.0.1 is starting in normal mode path=/var/lib/lxd
INFO[09-11|20:56:34] Kernel uid/gid map:
INFO[09-11|20:56:34] - u 0 0 4294967295
INFO[09-11|20:56:34] - g 0 0 4294967295
INFO[09-11|20:56:34] Configured LXD uid/gid map:
INFO[09-11|20:56:34] - u 0 100000 65536
INFO[09-11|20:56:34] - g 0 100000 65536
WARN[09-11|20:56:34] CGroup memory swap accounting is disabled, swap limits will be ignored.
INFO[09-11|20:56:34] Initializing local database
EROR[09-11|20:56:34] Failed to start the daemon: Both legacy and new local database files exists
INFO[09-11|20:56:34] Starting shutdown sequence
DBUG[09-11|20:56:34] Not unmounting temporary filesystems (containers are still running)
INFO[09-11|20:56:34] Saving simplestreams cache
INFO[09-11|20:56:34] Saved simplestreams cache
Error: Both legacy and new local database files exists

Sounds like you do have a /var/lib/lxd/lxd.db file now, possibly created by trying to run the first command I gave you.

If you do, can you show sqlite3 /var/lib/lxd/lxd.db .dump? If that’s empty, then go ahead and remove that file, then try running lxd --debug --group lxd again.

root@lxdserver01:/var/lib/lxd# ls -al
total 12
drwxr-xr-x 13 lxd nogroup 251 Sep 11 20:56 .
drwxr-xr-x 38 root root 4096 Sep 4 21:31 

drwx–x--x 2 root root 22 Sep 5 22:21 containers
drwx------ 3 root root 36 Sep 4 21:42 database
drwx–x--x 3 root root 22 Sep 5 22:21 devices
drwxr-xr-x 2 root root 6 Sep 4 21:42 devlxd
drwx------ 2 root root 6 Sep 4 21:42 disks
drwx------ 2 root root 157 Sep 5 22:20 images
-rw-r–r-- 1 root root 0 Sep 10 23:23 lxd.db
drwx–x--x 2 root root 6 Sep 4 21:42 networks
drwx------ 4 root root 37 Sep 5 22:21 security
-rw-r–r-- 1 root root 1903 Sep 4 21:42 server.crt
-rw------- 1 root root 3243 Sep 4 21:42 server.key
drwx–x--x 2 root root 6 Sep 4 21:42 shmounts
drwx------ 2 root root 6 Sep 4 21:42 snapshots
drwx–x--x 4 root root 36 Sep 5 22:14 storage-pools
srw-rw---- 1 root lxd 0 Sep 11 20:56 unix.socket
root@lxdserver01:/var/lib/lxd# sqlite3 /var/lib/lxd/lxd.db .dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
COMMIT;
root@lxdserver01:/var/lib/lxd# rm lxd.db
root@lxdserver01:/var/lib/lxd#
root@lxdserver01:/var/lib/lxd# systemctl restart lxd lxd.socket

System hangs on the systemctl command and here is the debug output.

root@lxdserver01:~# lxd --debug --group lxd
DBUG[09-11|21:40:12] Connecting to a local LXD over a Unix socket
DBUG[09-11|21:40:12] Sending request to LXD etag= method=GET url=http://unix.socket/1.0
INFO[09-11|21:40:36] LXD 3.0.1 is starting in normal mode path=/var/lib/lxd
INFO[09-11|21:40:36] Kernel uid/gid map:
INFO[09-11|21:40:36] - u 0 0 4294967295
INFO[09-11|21:40:36] - g 0 0 4294967295
INFO[09-11|21:40:36] Configured LXD uid/gid map:
INFO[09-11|21:40:36] - u 0 100000 65536
INFO[09-11|21:40:36] - g 0 100000 65536
WARN[09-11|21:40:36] CGroup memory swap accounting is disabled, swap limits will be ignored.
INFO[09-11|21:40:36] Initializing local database
INFO[09-11|21:40:36] Initializing database gateway
INFO[09-11|21:40:36] Start database node address= id=1
INFO[09-11|21:40:36] Raft: Restored from snapshot 1-762-1536610622008
INFO[09-11|21:40:36] Raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:0}]
INFO[09-11|21:40:36] Raft: Node at 0 [Leader] entering Leader state
INFO[09-11|21:40:36] LXD isn’t socket activated
DBUG[09-11|21:40:36] Connecting to a local LXD over a Unix socket
DBUG[09-11|21:40:36] Sending request to LXD etag= method=GET url=http://unix.socket/1.0

And the status of the systemctl lxd and lxd.socket output

indiana@lxdserver01:~$ systemctl status lxd
● lxd.service - LXD - main daemon
Loaded: loaded (/lib/systemd/system/lxd.service; indirect; vendor preset: enabled)
Active: activating (start-post) (Result: exit-code) since Tue 2018-09-11 21:40:36 +03; 4min 35s ago
Docs: man:lxd(1)
Process: 2498 ExecStart=/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log (code=exited, status=1/FAILURE)
Process: 2481 ExecStartPre=/usr/lib/x86_64-linux-gnu/lxc/lxc-apparmor-load (code=exited, status=0/SUCCESS)
Main PID: 2498 (code=exited, status=1/FAILURE); Control PID: 2499 (lxd)
Tasks: 7
CGroup: /system.slice/lxd.service
└─2499 /usr/lib/lxd/lxd waitready --timeout=600

Sep 11 21:40:36 lxdserver01 systemd[1]: Starting LXD - main daemon

Sep 11 21:40:36 lxdserver01 lxd[2498]: lvl=warn msg=“CGroup memory swap accounting is disabled, swap limits will be ignored.” t=2018-09-11T21:40:36+0300
Sep 11 21:40:41 lxdserver01 lxd[2498]: lvl=eror msg=“Failed to start the daemon: failed to create raft factory: failed to create bolt store for raft logs: timeout” t=2018-09-11T21:40:41+0300
Sep 11 21:40:41 lxdserver01 lxd[2498]: Error: failed to create raft factory: failed to create bolt store for raft logs: timeout
Sep 11 21:40:41 lxdserver01 systemd[1]: lxd.service: Main process exited, code=exited, status=1/FAILURE
indiana@lxdserver01:~$
indiana@lxdserver01:~$ systemctl status lxd.socket
● lxd.socket - LXD - unix socket
Loaded: loaded (/lib/systemd/system/lxd.socket; enabled; vendor preset: enabled)
Active: active (running) since Tue 2018-09-11 21:40:36 +03; 4min 52s ago
Docs: man:lxd(1)
Listen: /var/lib/lxd/unix.socket (Stream)
Tasks: 0 (limit: 4504)
CGroup: /system.slice/lxd.socket

Sep 11 21:40:36 lxdserver01 systemd[1]: Starting LXD - unix socket.
Sep 11 21:40:36 lxdserver01 systemd[1]: Listening on LXD - unix socket.

So systemd keeps getting in the way, can you do:

  • systemctl stop lxd.socket lxd.service
  • lxd --debug --group lxd

And don’t try to manually start/restart the systemd units until we’ve resolved this.

Humm, ok.

Output of the systemctl stop lxd.socket lxd.service

root@lxdserver01:~# lxd --debug --group lxd
DBUG[09-11|21:49:19] Connecting to a local LXD over a Unix socket
DBUG[09-11|21:49:19] Sending request to LXD etag= method=GET url=http://unix.socket/1.0
INFO[09-11|21:50:07] LXD 3.0.1 is starting in normal mode path=/var/lib/lxd
INFO[09-11|21:50:07] Kernel uid/gid map:
INFO[09-11|21:50:07] - u 0 0 4294967295
INFO[09-11|21:50:07] - g 0 0 4294967295
INFO[09-11|21:50:07] Configured LXD uid/gid map:
INFO[09-11|21:50:07] - u 0 100000 65536
INFO[09-11|21:50:07] - g 0 100000 65536
WARN[09-11|21:50:07] CGroup memory swap accounting is disabled, swap limits will be ignored.
INFO[09-11|21:50:07] Initializing local database
INFO[09-11|21:50:07] Initializing database gateway
INFO[09-11|21:50:07] Start database node address= id=1
INFO[09-11|21:50:07] Raft: Restored from snapshot 1-762-1536610622008
INFO[09-11|21:50:07] Raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:0}]
INFO[09-11|21:50:07] Raft: Node at 0 [Leader] entering Leader state
INFO[09-11|21:50:07] LXD isn’t socket activated
DBUG[09-11|21:50:07] Connecting to a local LXD over a Unix socket
DBUG[09-11|21:50:07] Sending request to LXD etag= method=GET url=http://unix.socket/1.0
DBUG[09-11|21:50:07] Detected stale unix socket, deleting
INFO[09-11|21:50:07] Starting /dev/lxd handler:
INFO[09-11|21:50:07] - binding devlxd socket socket=/var/lib/lxd/devlxd/sock
INFO[09-11|21:50:07] REST API daemon:
INFO[09-11|21:50:07] - binding Unix socket socket=/var/lib/lxd/unix.socket
INFO[09-11|21:50:07] Initializing global database
INFO[09-11|21:50:07] Initializing storage pools
DBUG[09-11|21:50:07] Initializing and checking storage pool “default”.
DBUG[09-11|21:50:07] Initializing a ZFS driver.
DBUG[09-11|21:50:07] Checking ZFS storage pool “default”.
DBUG[09-11|21:50:07] ZFS storage pool “default” does not exist. Trying to import it.
EROR[09-11|21:50:07] Failed to start the daemon: ZFS storage pool “default” could not be imported: cannot import ‘default’: no such pool available

INFO[09-11|21:50:07] Starting shutdown sequence
INFO[09-11|21:50:07] Stopping REST API handler:
INFO[09-11|21:50:07] - closing socket socket=/var/lib/lxd/unix.socket
INFO[09-11|21:50:07] Stopping /dev/lxd handler
INFO[09-11|21:50:07] - closing socket socket=/var/lib/lxd/devlxd/sock
INFO[09-11|21:50:07] Closing the database
INFO[09-11|21:50:07] Stop database gateway
INFO[09-11|21:50:07] Stop raft instance
INFO[09-11|21:50:08] Raft: Starting snapshot up to 776
INFO[09-11|21:50:08] Raft: Compacting logs from 635 to 648
INFO[09-11|21:50:08] Raft: Snapshot to 776 complete
INFO[09-11|21:50:08] Stopping REST API handler:
INFO[09-11|21:50:08] Stopping /dev/lxd handler
INFO[09-11|21:50:08] Stopping REST API handler:
INFO[09-11|21:50:08] Stopping /dev/lxd handler
INFO[09-11|21:50:08] Unmounting temporary filesystems
INFO[09-11|21:50:08] Done unmounting temporary filesystems
INFO[09-11|21:50:08] Saving simplestreams cache
INFO[09-11|21:50:08] Saved simplestreams cache
Error: ZFS storage pool “default” could not be imported: cannot import ‘default’: no such pool available

Ok, lets go with a quick and dirty workaround to get LXD running, then we can fix its database.

  • truncate -s 1G /tmp/zfs.img
  • zpool create default /tmp/zfs.img -m none
  • lxd --debug --group lxd

LXD should then start fine, at which point, from another terminal, please run lxd sql global "SELECT * FROM storage_pools_config;"

1 Like
  • truncate -s 1G /tmp/zfs.img
  • zpool create default /tmp/zfs.img -m none

Output

root@lxdserver01:~# lxd --debug --group lxd
INFO[09-11|21:59:05] LXD 3.0.1 is starting in normal mode path=/var/lib/lxd
INFO[09-11|21:59:05] Kernel uid/gid map:
INFO[09-11|21:59:05] - u 0 0 4294967295
INFO[09-11|21:59:05] - g 0 0 4294967295
INFO[09-11|21:59:05] Configured LXD uid/gid map:
INFO[09-11|21:59:05] - u 0 100000 65536
INFO[09-11|21:59:05] - g 0 100000 65536
WARN[09-11|21:59:05] CGroup memory swap accounting is disabled, swap limits will be ignored.
INFO[09-11|21:59:05] Initializing local database
INFO[09-11|21:59:05] Initializing database gateway
INFO[09-11|21:59:05] Start database node address= id=1
INFO[09-11|21:59:05] Raft: Restored from snapshot 1-776-1536691808014
INFO[09-11|21:59:05] Raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:0}]
INFO[09-11|21:59:05] Raft: Node at 0 [Leader] entering Leader state
INFO[09-11|21:59:05] LXD isn’t socket activated
INFO[09-11|21:59:05] Starting /dev/lxd handler:
INFO[09-11|21:59:05] - binding devlxd socket socket=/var/lib/lxd/devlxd/sock
INFO[09-11|21:59:05] REST API daemon:
INFO[09-11|21:59:05] - binding Unix socket socket=/var/lib/lxd/unix.socket
INFO[09-11|21:59:05] Initializing global database
INFO[09-11|21:59:05] Initializing storage pools
DBUG[09-11|21:59:05] Initializing and checking storage pool “default”.
DBUG[09-11|21:59:05] Initializing a ZFS driver.
DBUG[09-11|21:59:05] Checking ZFS storage pool “default”.
DBUG[09-11|21:59:05] Initializing and checking storage pool “lxdpool”.
DBUG[09-11|21:59:05] Initializing a ZFS driver.
DBUG[09-11|21:59:05] Checking ZFS storage pool “lxdpool”.
DBUG[09-11|21:59:05] Initializing a ZFS driver.
INFO[09-11|21:59:05] Initializing networks
INFO[09-11|21:59:05] Loading configuration
DBUG[09-11|21:59:05] Initialized inotify with file descriptor 13
INFO[09-11|21:59:05] Pruning expired images
INFO[09-11|21:59:05] Done pruning expired images
INFO[09-11|21:59:05] Expiring log files
INFO[09-11|21:59:05] Updating instance types
INFO[09-11|21:59:05] Updating images
INFO[09-11|21:59:05] Done expiring log files
DBUG[09-11|21:59:05] Processing image alias=centos/7 fp=f6b2cd8bc731038b3a8b0eb0f4f8b668815945d1f41dedab119f3c2e7a0f4aa6 protocol=simplestreams server=https://images.linuxcontainers.org
DBUG[09-11|21:59:05] Connecting to a remote simplestreams server
DBUG[09-11|21:59:05] Initializing a ZFS driver.
WARN[09-11|21:59:05] Unable to update backup.yaml at this time. name=centos01
DBUG[09-11|21:59:05] Mounting ZFS storage volume for container “centos01” on storage pool “lxdpool”
DBUG[09-11|21:59:05] Mounted ZFS storage volume for container “centos01” on storage pool “lxdpool”
DBUG[09-11|21:59:05] Unmounting ZFS storage volume for container “centos01” on storage pool “lxdpool”.
DBUG[09-11|21:59:05] Unmounted ZFS storage volume for container “centos01” on storage pool “lxdpool”.
DBUG[09-11|21:59:06] Mounting ZFS storage volume for container “centos01” on storage pool “lxdpool”
DBUG[09-11|21:59:06] Mounted ZFS storage volume for container “centos01” on storage pool “lxdpool”
INFO[09-11|21:59:06] Starting container action=start created=2018-09-05T22:21:07+0300 ephemeral=false name=centos01 stateful=false used=2018-09-05T22:21:19+0300
DBUG[09-11|21:59:06] handling ip=@ method=GET url=/1.0
DBUG[09-11|21:59:06] handling ip=@ method=GET url=/internal/containers/1/onstart
DBUG[09-11|21:59:06] Initializing a ZFS driver.
DBUG[09-11|21:59:06] Mounting ZFS storage volume for container “centos01” on storage pool “lxdpool”
DBUG[09-11|21:59:06] Mounted ZFS storage volume for container “centos01” on storage pool “lxdpool”
DBUG[09-11|21:59:06] Scheduler: container centos01 started: re-balancing
DBUG[09-11|21:59:06] Scheduler: network: vethETMFQW has been added: updating network priorities
INFO[09-11|21:59:06] Started container action=start created=2018-09-05T22:21:07+0300 ephemeral=false name=centos01 stateful=false used=2018-09-05T22:21:19+0300
DBUG[09-11|21:59:06] Scheduler: network: veth59CD14 has been added: updating network priorities
DBUG[09-11|21:59:07] Database error: &errors.errorString{s:“sql: no rows in result set”}
INFO[09-11|21:59:07] Downloading image alias=centos/7 server=https://images.linuxcontainers.org
INFO[09-11|21:59:08] Done updating instance types
DBUG[09-11|21:59:55] Database error: &errors.errorString{s:“sql: no rows in result set”}
DBUG[09-11|21:59:55] Initializing a ZFS driver.
DBUG[09-11|21:59:55] Creating ZFS storage volume for image “02deea70bf3ce6edd484b44e77a43e07b79e09338893d1a339beeff68a88f438” on storage pool “lxdpool”.
DBUG[09-11|22:00:00] Created ZFS storage volume for image “02deea70bf3ce6edd484b44e77a43e07b79e09338893d1a339beeff68a88f438” on storage pool “lxdpool”.
INFO[09-11|22:00:00] Image downloaded alias=centos/7 server=https://images.linuxcontainers.org
DBUG[09-11|22:00:00] Initializing a ZFS driver.
DBUG[09-11|22:00:00] Deleting ZFS storage volume for image “f6b2cd8bc731038b3a8b0eb0f4f8b668815945d1f41dedab119f3c2e7a0f4aa6” on storage pool “lxdpool”.
DBUG[09-11|22:00:01] Deleted ZFS storage volume for image “f6b2cd8bc731038b3a8b0eb0f4f8b668815945d1f41dedab119f3c2e7a0f4aa6” on storage pool “lxdpool”.
INFO[09-11|22:00:01] Done updating images

But nothing comes out, when I execute the command
root@lxdserver01:~# lxd sql global “SELECT * FROM storage_pools_config;”

But when I stop the debug, lxd sql replied that error.
root@lxdserver01:~# lxd sql global “SELECT * FROM storage_pools_config;”
Error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: no such file or directory

root@lxdserver01:~# systemctl status lxd.socket
● lxd.socket - LXD - unix socket
Loaded: loaded (/lib/systemd/system/lxd.socket; enabled; vendor preset: enabled)
Active: active (running) since Tue 2018-09-11 22:04:26 +03; 9min ago
Docs: man:lxd(1)
Listen: /var/lib/lxd/unix.socket (Stream)
Tasks: 0 (limit: 4504)
CGroup: /system.slice/lxd.socket

Sep 11 22:04:26 lxdserver01 systemd[1]: Starting LXD - unix socket.
Sep 11 22:04:26 lxdserver01 systemd[1]: Listening on LXD - unix socket.

Yeah, that’s normal, lxd sql requires LXD to be running, if you stop it, it won’t work anymore.

Can you show lxd sql global "SELECT * FROM storage_pools;"?

Strange, cos when I execute the lxd sql command gives socket error and checked that no unix.socket at the /var/lib/lxd directory. Anyway i restarted the lxd.socket and gives the following output.
root@lxdserver01:/var/lib/lxd# lxd sql global “SELECT * FROM storage_pools;”
±—±--------±-------±------------±------+
| id | name | driver | description | state |
±—±--------±-------±------------±------+
| 3 | default | zfs | | 1 |
| 4 | lxdpool | zfs | | 1 |
±—±--------±-------±------------±------+

Can you show lxc storage show lxdpool and lxc storage show default?

root@lxdserver01:~# lxc storage show lxdpool
config:
source: lxdpool
volatile.initial_source: /dev/sdb
zfs.pool_name: lxdpool
description: “”
name: lxdpool
driver: zfs
used_by:

  • /1.0/containers/centos01
  • /1.0/images/02deea70bf3ce6edd484b44e77a43e07b79e09338893d1a339beeff68a88f438
  • /1.0/profiles/default
    status: Created
    locations:
  • none
    root@lxdserver01:~#
    root@lxdserver01:~# lxc storage show default
    config:
    source: default
    volatile.initial_source: /dev/sdb
    zfs.pool_name: default
    description: “”
    name: default
    driver: zfs
    used_by: []
    status: Created
    locations:
  • none