LXD hangs during apt upgrade on Mint 17 (ubuntu 16.04)

LXD
1 
Setting up lxd (2.21-0ubuntu2~16.04.1) ...
Job for lxd-containers.service failed because a timeout was exceeded. See "systemctl status lxd-containers.service" and "journalctl -xe" for details.
lxd-containers.service couldn't start.

jwl@blackie2:~$ sudo systemctl status lxd-containers.service
● lxd-containers.service - LXD - container startup/shutdown
   Loaded: loaded (/lib/systemd/system/lxd-containers.service; enabled; vendor preset: enabled)
   Active: activating (start) since Wed 2018-01-31 16:01:57 EST; 35s ago
     Docs: man:lxd(1)
 Main PID: 28980 (lxd)
   CGroup: /system.slice/lxd-containers.service
           └─28980 /usr/bin/lxd activateifneeded

Jan 31 16:01:57 blackie2 systemd[1]: Starting LXD - container startup/shutdown...
jwl@blackie2:~$ journalctl -xe
Jan 31 16:01:57 blackie2 lxd[29007]: lvl=warn msg="Failed to set new ZFS mountpoint: cannot open 'lxd1/containers/pine': dataset does not exis
Jan 31 16:01:57 blackie2 lxd[29007]: lvl=warn msg="Storage volumes database already contains an entry for the image." t=2018-01-31T16:01:57-05
Jan 31 16:01:57 blackie2 lxd[29007]: lvl=warn msg="Failed to set new ZFS mountpoint: cannot open 'lxd1/images/2cab90c0c342346ea154bc2e8cacdae7
Jan 31 16:01:57 blackie2 lxd[29007]: error: containers/first: Failed to set new zfs mountpoint: Failed to run: zfs set mountpoint=/var/lib/lxd
Jan 31 16:01:57 blackie2 lxd[29007]: containers/pine: Failed to set new zfs mountpoint: Failed to run: zfs set mountpoint=/var/lib/lxd/storage
Jan 31 16:01:57 blackie2 systemd[1]: lxd.service: Main process exited, code=exited, status=1/FAILURE
Jan 31 16:02:15 blackie2 systemd[1]: Reloading.
Jan 31 16:02:15 blackie2 systemd[1]: Started CUPS Scheduler.
-- Subject: Unit cups.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit cups.service has finished starting up.
-- 
-- The start-up result is done.
Jan 31 16:02:15 blackie2 systemd[1]: Started ACPI event daemon.
-- Subject: Unit acpid.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit acpid.service has finished starting up.
-- 
-- The start-up result is done.
Jan 31 16:02:20 blackie2 sudo[26486]: pam_unix(sudo:session): session closed for user root
Jan 31 16:02:32 blackie2 sudo[6336]:      jwl : TTY=pts/7 ; PWD=/home/jwl ; USER=root ; COMMAND=/bin/systemctl status lxd-containers.service
Jan 31 16:02:32 blackie2 sudo[6336]: pam_unix(sudo:session): session opened for user root by jwl(uid=0)
Jan 31 16:02:32 blackie2 sudo[6336]: pam_unix(sudo:session): session closed for user root

zfs output

sudo zfs list
no datasets available
lxc version
2.21



I don't know what other information would be of interest.

Thanks,
Jim.
2

Can you show what LXD configuration is there for storage?

lxc storage list

3

lxc storage list hangs. It just sits there blinking a cursor until I hit ^C

4

FYI I kept running apt upgrade and dpkg --configure -a until I got a clean upgrade but lxd is no longer working.

5

Okay.
In a subdirectory in /var/lib/lxd/ there should be a file lxd.db. And alongside, an older backup of that file.
They are SQLite databases, so you can read them with sqlitebrowser. With this utility, you can browse the data of the lxd.db file and particularly look into the storage tables. In that way, you can get an idea of what lxd expects to find in the storage.
There is a chance that lxd.db is corrupted and needs fixing, or replacement with the backup.

The fact that sudo zpool list and sudo zfs list does not give you any output, is not good. But let’s see what LXD thinks about the storage settings.

6

Very interesting. There are no storage tables in the lxd.db.bak file but there are in the lxd.db.

Here’s a dump of the storage* tables;

root@blackie2:/var/lib/lxd# sqlite3 lxd.db
SQLite version 3.11.0 2016-02-15 17:29:24
Enter ".help" for usage hints.
sqlite> 
sqlite> select * from storage_pools;
1|lxd1|zfs|
sqlite> select * from storage_pools_config;
223|1|size|20GB
224|1|source|lxd1
sqlite> select * from storage_volumes;
1|first|1|0|
2|pine|1|0|
3|2cab90c0c342346ea154bc2e8cacdae752a70747a755ce1f2970c9a9ebb5fe8c|1|1|
sqlite> select * from storage_volumes_config;

Also

root@blackie2:/var/lib/lxd# zpool list
no pools available
root@blackie2:/var/lib/lxd# zfs list
no datasets available
7

What I get on a computer, is this:

sqlite> select * from storage_pools_config;
2|1|size|30GB
3|1|source|/var/snap/lxd/common/lxd/disks/default.img
4|1|zfs.pool_name|default
sqlite>

(I selected the multiline text above, then clicked on the </> icon. It indents it a bit and makes it to a fixed font. While writing this, I notice that you found it as well).

The important line, is the source line. In my case, it’s a loopback file that has a ZFS filesystem.
In your case, it is a partition, isn’t it?
You would first need to identify the partition that you used, and then try to get it to get recognized by ZFS. Once that is done, then LXD would try to pick it up.

8

It’s a partiton formatted as zfs. I’m new to zfs. I’ll have to investigate things. I tried to paste an image here but it wouldn’t let me.
Zfs partiton

Are you saying something would generate the storage_pools_config table or is that something I would have to construct?

9

I think that lxd.db is fine. It has the correct information that, when the ZFS partition comes alive again, LXD will manage to work (or report explicitly if it encounters any issues).

Your goal would be to figure out how to get the ZFS partition come alive. There might be some diagnostic tools or procedures to diagnose what could be happening. This would be solely ZFS investigatory work.

10

Think I’ll blow lxd away and reinstall it. I created a zfs pool but nothing changed.

11

Thanks for the help.