Lxd daemon startup problem

schoenbeck · November 18, 2018, 11:02pm

Hello,

after updating my ubuntu 18.04. installation, the lxd daemon is not starting up again and no container is being started. The log says “Error: failed to prepare loop device: no such file or directory”. Maybe it has something to do with the strorage pool “lvm” which I created during installation and tried to delete without success after the VG was deleted?!
Can someone help me fix this? Thank you very much!

lxd --debug --group lxd

INFO[11-18|23:56:30] LXD 3.0.2 is starting in normal mode path=/var/lib/lxd
INFO[11-18|23:56:30] Kernel uid/gid map:
INFO[11-18|23:56:30] - u 0 0 4294967295
INFO[11-18|23:56:30] - g 0 0 4294967295
INFO[11-18|23:56:30] Configured LXD uid/gid map:
INFO[11-18|23:56:30] - u 0 100000 65536
INFO[11-18|23:56:30] - g 0 100000 65536
WARN[11-18|23:56:30] CGroup memory swap accounting is disabled, swap limits will be ignored.
INFO[11-18|23:56:30] Initializing local database
INFO[11-18|23:56:30] Initializing database gateway
INFO[11-18|23:56:30] Start database node address= id=1
INFO[11-18|23:56:30] Raft: Restored from snapshot 1-742-1542568879099
INFO[11-18|23:56:30] Raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:0}]
INFO[11-18|23:56:30] Raft: Node at 0 [Leader] entering Leader state
INFO[11-18|23:56:30] Dqlite: starting event loop
DBUG[11-18|23:56:30] Dqlite: accepting connections
INFO[11-18|23:56:30] LXD isn’t socket activated
INFO[11-18|23:56:30] Starting /dev/lxd handler:
INFO[11-18|23:56:30] - binding devlxd socket socket=/var/lib/lxd/devlxd/sock
INFO[11-18|23:56:30] REST API daemon:
INFO[11-18|23:56:30] - binding Unix socket socket=/var/lib/lxd/unix.socket
INFO[11-18|23:56:30] Initializing global database
INFO[11-18|23:56:30] Dqlite: handling new connection (fd=18)
INFO[11-18|23:56:30] Dqlite: connected address=0 attempt=0
INFO[11-18|23:56:30] Initializing storage pools
DBUG[11-18|23:56:30] Initializing and checking storage pool “default”
DBUG[11-18|23:56:30] Checking BTRFS storage pool “default”
DBUG[11-18|23:56:30] Initializing and checking storage pool “fast”
DBUG[11-18|23:56:30] Checking BTRFS storage pool “fast”
DBUG[11-18|23:56:30] Initializing and checking storage pool “lvm”
DBUG[11-18|23:56:30] Checking LVM storage pool “lvm”
DBUG[11-18|23:56:30] Checked LVM storage pool “lvm”
DBUG[11-18|23:56:30] Initializing and checking storage pool “lvm2”
DBUG[11-18|23:56:30] Checking LVM storage pool “lvm2”
EROR[11-18|23:56:30] Failed to start the daemon: failed to prepare loop device: no such file or directory
INFO[11-18|23:56:30] Starting shutdown sequence
INFO[11-18|23:56:30] Stopping REST API handler:
INFO[11-18|23:56:30] - closing socket socket=/var/lib/lxd/unix.socket
INFO[11-18|23:56:30] Stopping /dev/lxd handler
INFO[11-18|23:56:30] - closing socket socket=/var/lib/lxd/devlxd/sock
INFO[11-18|23:56:30] Closing the database
INFO[11-18|23:56:30] Dqlite: closing client
INFO[11-18|23:56:30] Stop database gateway
INFO[11-18|23:56:30] Stop raft instance
INFO[11-18|23:56:30] Dqlite: stopping event loop
INFO[11-18|23:56:30] Dqlite: event loop stopped
INFO[11-18|23:56:30] Stopping REST API handler:
INFO[11-18|23:56:30] Stopping /dev/lxd handler
INFO[11-18|23:56:30] Stopping REST API handler:
INFO[11-18|23:56:30] Stopping /dev/lxd handler
INFO[11-18|23:56:30] Unmounting temporary filesystems
INFO[11-18|23:56:31] Done unmounting temporary filesystems
INFO[11-18|23:56:31] Saving simplestreams cache
INFO[11-18|23:56:31] Saved simplestreams cache
Error: failed to prepare loop device: no such file or directory

stgraber · November 19, 2018, 1:07am

Yeah, LXD seems to be expecting a /var/lib/lxd/disks/lvm2.img file which doesn’t exist.

I suspect you may be able to get it past this error with:

truncate -s 1G /var/lib/lxd/disks/lvm2.img
pvcreate /var/lib/lxd/disks/lvm2.img

If it complains about the VG then, you can create that VG with that newly created PV, which should get LXD online, at which point you can delete that storage pool with lxc storage delete lvm2

schoenbeck · November 19, 2018, 3:30am

Thank you very much, that helped! It worked out with

truncate -s 1G /var/lib/lxd/disks/lvm2.img
losetup /dev/loop0 /var/lib/lxd/disks/lvm2.img
pvcreate /dev/loop0
vgcreate lvm2 /dev/loop0

I will try to delete storage pools lvm and lvm2 which are only remnants from playing around with storage pools during installation…

schoenbeck · November 19, 2018, 3:45pm

These storage pools can’t be deleted:

    ~ # lxc storage delete lvm2
    Error: storage pool "lvm2" has volumes attached to it
    ~ # lxc storage delete lvm
    Error: storage pool "lvm" has volumes attached to it

~ # lxc storage show lvm2
config:
  lvm.thinpool_name: LXDThinPool
  lvm.vg_name: lvm2
  size: 15GB
  source: /var/lib/lxd/disks/lvm2.img
description: ""
name: lvm2
driver: lvm
used_by:
- /1.0/containers/office
status: Created
locations:
- none

Unfortunately the container “office” doesn’t exist anymore. Is there a better way to delete the storage pools? Thank you!

stgraber · November 19, 2018, 4:19pm

What does lxc storage volume list lvm2 get you?

If it’s a leftover container storage volume we’ll likely need a tiny bit of database surgery to fix it.

lxd sql global "SELECT * FROM storage_volumes;" should help figure out what needs cleaning.

schoenbeck · November 19, 2018, 4:28pm

~ # lxc storage volume list lvm2
+-----------+--------+-------------+---------+
|   TYPE    |  NAME  | DESCRIPTION | USED BY |
+-----------+--------+-------------+---------+
| container | office |             | 1       |
+-----------+--------+-------------+---------+

~ # lxc storage volume list lvm
+-----------+--------+-------------+---------+
|   TYPE    |  NAME  | DESCRIPTION | USED BY |
+-----------+--------+-------------+---------+
| container | office |             | 1       |
+-----------+--------+-------------+---------+


~ # lxd sql global "SELECT * FROM storage_volumes;"
+----+--------+-----------------+---------+------+-------------+
| id |  name  | storage_pool_id | node_id | type | description |
+----+--------+-----------------+---------+------+-------------+
| 1  | www    | 2               | 1       | 0    |             |
| 3  | db     | 3               | 1       | 0    |             |
| 5  | intern | 2               | 1       | 0    |             |
| 10 | office | 5               | 1       | 0    |             |
| 12 | office | 6               | 1       | 0    |             |
| 21 | chat   | 2               | 1       | 0    |             |
+----+--------+-----------------+---------+------+-------------+

stgraber · November 19, 2018, 5:17pm

Ok, can you also show:

lxc list
lxd sql global “SELECT * FROM storage_pools;”

schoenbeck · November 19, 2018, 6:32pm

~ # lxc list
+--------+---------+----------------------+-----------------------------------------------+------------+-----------+
|  NAME  |  STATE  |         IPV4         |                     IPV6                      |    TYPE    | SNAPSHOTS |
+--------+---------+----------------------+-----------------------------------------------+------------+-----------+
| chat   | RUNNING | 10.210.16.160 (eth0) | fd42:b039:472d:1795:216:3eff:feaa:eaf (eth0)  | PERSISTENT | 0         |
+--------+---------+----------------------+-----------------------------------------------+------------+-----------+
| db     | RUNNING | 10.210.16.173 (eth0) | fd42:b039:472d:1795:216:3eff:fe8c:302a (eth0) | PERSISTENT | 0         |
+--------+---------+----------------------+-----------------------------------------------+------------+-----------+
| intern | RUNNING | 10.210.16.241 (eth0) | fd42:b039:472d:1795:216:3eff:fe8d:7a34 (eth0) | PERSISTENT | 0         |
+--------+---------+----------------------+-----------------------------------------------+------------+-----------+
| www    | RUNNING | 10.210.16.27 (eth0)  | fd42:b039:472d:1795:216:3eff:fef5:5b80 (eth0) | PERSISTENT | 0         |
+--------+---------+----------------------+-----------------------------------------------+------------+-----------+

~ # lxd sql global "SELECT * FROM storage_pools;"
+----+---------+--------+-------------+-------+
| id |  name   | driver | description | state |
+----+---------+--------+-------------+-------+
| 2  | default | btrfs  |             | 1     |
| 3  | fast    | btrfs  |             | 1     |
| 5  | lvm     | lvm    |             | 1     |
| 6  | lvm2    | lvm    |             | 1     |
| 8  | lvm3    | lvm    |             | 1     |
+----+---------+--------+-------------+-------+

From which only default and fast are in use.

stgraber · November 21, 2018, 4:59am

Ok, if you’re sure you have nothing using them and lxc storage delete doesn’t work to get rid of them, you can use:

lxd sql global "DELETE FROM storage_pools WHERE id IN (5,6,8);"

Which should get rid of all 3 of them. You may then want to cleanup any loop file that’s left for them in /var/lib/lxd/disks/.

schoenbeck · December 6, 2018, 12:33am

That worked, thank you for your help!
(Sorry for the long delay, I waited for the next reboot to confirm this.)