Lxd 4.1: even more ceph issues

1

It’s me again, this time with a fully fresh ceph and lxd environment.

Tried to add a storage pool and I’m receiving this error creating a container and the pool is shown as status: Pending:

root@aa1-cptef102-n4:/home/ubuntu# lxc launch --storage ceph ubuntu:20.04
Creating the instance
Error: Failed instance creation: Create instance from image: Failed to create mount directory "/var/snap/lxd/common/lxd/storage-pools/ceph/images/42424696bc6fc56ebff11508b597946b8f451f002a780bf0a943051130712a8f": mkdir /var/snap/lxd/common/lxd/storage-pools/ceph/images/42424696bc6fc56ebff11508b597946b8f451f002a780bf0a943051130712a8f: no such file or directory

Here’s the pool
config:
ceph.osd.pool_name: lxd-rootfs-01
ceph.user.name: lxd
description: “”
name: ceph
driver: ceph
used_by: []
status: Pending
locations:
- aa1-cptef101-n1
- aa1-cptef101-n2
- aa1-cptef101-n3
- aa1-cptef101-n4
- aa1-cptef102-n1
- aa1-cptef102-n2
- aa1-cptef102-n3
- aa1-cptef102-n4
The pool is visible:
info:
description: “”
driver: ceph
name: ceph
space used: 0B
total space: 60.80TB
used by: {}

The pool was created as such:
lxc storage create ceph ceph source=lxd-rootfs-01 --target aa1-cptef101-n1
lxc storage set ceph ceph.user.name=lxd
lxc storage set ceph ceph.osd.pool_name=lxd-rootfs-01

2

That’s a new one… Can you show ls -lh /var/snap/lxd/common/lxd/storage-pools/ceph/?

3

No such file or directory on all 8 nodes.

4

Oh, hold on, that pool hasn’t been fully created yet, it still shows as pending.

Can you do lxc storage create ceph ceph to finalize its creation?

5

Got it - had to add ceph.user.name=lxd as well:

root@aa1-cptef102-n3:/home/ubuntu# lxc storage create ceph ceph ceph.user.name=lxd
Storage pool ceph created

Looks like creating containers works now, too:

root@aa1-cptef102-n3:/home/ubuntu# lxc launch ubuntu:20.04 --storage ceph
Creating the instance
Instance name is: legible-garfish

The instance you are starting doesn't have any network attached to it.
  To create a new network, use: lxc network create
  To attach a network to an instance, use: lxc network attach

Starting legible-garfish

Thanks!

6

@freeekanayaka can we do something to filter out pending pools and networks in most operations?

In this case it looks like LXD completely ignored the fact that the pool hasn’t actually been fully created and happily allowed it to be used to store stuff.

7

I thought it was already the case. I’ll look into that.

8

Re-using thread as it’s a similar issue:

Trying to create a second pool, but it seems to be trying the wrong user?

ubuntu@aa1-cptef101-n1:~$ lxc storage show ceph-vault
config:
  ceph.osd.pool_name: lxd-virtualvaultfs-01
  ceph.user.name: lxd
description: ""
name: ceph-vault
driver: ceph
used_by: []
status: Pending
locations:
- aa1-cptef101-n1
- aa1-cptef101-n2
- aa1-cptef101-n3
- aa1-cptef101-n4
- aa1-cptef102-n1
- aa1-cptef102-n2
- aa1-cptef102-n3
- aa1-cptef102-n4


ubuntu@aa1-cptef101-n1:~$ lxc storage create ceph-vault ceph
Error: Failed to run: ceph --name client.admin --cluster ceph osd pool create lxd-virtualvaultfs-01 32: 2020-06-16 17:59:21.589552 7ff80a18e700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-06-16 17:59:21.590811 7ff80a18e700 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication
2020-06-16 17:59:21.590816 7ff80a18e700  0 librados: client.admin authentication error (95) Operation not supported
[errno 95] error connecting to the cluster

The pool was created as such:

lxc storage create ceph-vault ceph source=lxd-virtualvaultfs-01 --target aa1-cptef101-n1
lxc storage create ceph-vault ceph source=lxd-virtualvaultfs-01 --target aa1-cptef101-n2
lxc storage create ceph-vault ceph source=lxd-virtualvaultfs-01 --target aa1-cptef101-n3
lxc storage create ceph-vault ceph source=lxd-virtualvaultfs-01 --target aa1-cptef101-n4
lxc storage create ceph-vault ceph source=lxd-virtualvaultfs-01 --target aa1-cptef102-n1
lxc storage create ceph-vault ceph source=lxd-virtualvaultfs-01 --target aa1-cptef102-n2
lxc storage create ceph-vault ceph source=lxd-virtualvaultfs-01 --target aa1-cptef102-n3
lxc storage create ceph-vault ceph source=lxd-virtualvaultfs-01 --target aa1-cptef102-n4
lxc storage set ceph-vault ceph.user.name=lxd
lxc storage set ceph-vault ceph.cluster_name=ceph
9

@stgraber any ideas? Sorry for ping.

10

Hmm, does it make a difference if you pass in those properties as part of the final lxc storage create?

lxc storage create ceph-vault ceph ceph.user.name=lxd ceph.cluster_name=ceph

11

Yes, but not in the expected way:

ubuntu@aa1-cptef101-n1:~$ lxc storage create ceph-vault ceph ceph.user.name=lxd ceph.cluster_name=ceph
Error: Post "http://unix.socket/1.0/storage-pools": EOF
ubuntu@aa1-cptef101-n1:~$ lxc storage create ceph-vault ceph ceph.user.name=lxd ceph.cluster_name=ceph
Error: Storage pool directory "/var/snap/lxd/common/lxd/storage-pools/ceph-vault" already exists

ubuntu@aa1-cptef101-n1:~$ lxc storage ls
+------------+-------------+--------+---------+---------+
|    NAME    | DESCRIPTION | DRIVER |  STATE  | USED BY |
+------------+-------------+--------+---------+---------+
| ceph       |             | ceph   | CREATED | 9       |
+------------+-------------+--------+---------+---------+
| ceph-vault |             | ceph   | PENDING | 0       |
+------------+-------------+--------+---------+---------+

Still shows as pending

12

Do rm -rf /var/snap/lxd/common/lxd/storage-pools/ceph-vault and try again.

The first error indicates that LXD just completely crashed, if that happens when you try again, we’ll have to figure out why that is.

13

Worked that time:

ubuntu@aa1-cptef101-n1:~$ lxc storage create ceph-vault ceph ceph.user.name=lxd ceph.cluster_name=ceph
Storage pool ceph-vault created
ubuntu@aa1-cptef101-n1:~$ lxc storage ls
+------------+-------------+--------+---------+---------+
|    NAME    | DESCRIPTION | DRIVER |  STATE  | USED BY |
+------------+-------------+--------+---------+---------+
| ceph       |             | ceph   | CREATED | 9       |
+------------+-------------+--------+---------+---------+
| ceph-vault |             | ceph   | CREATED | 0       |
+------------+-------------+--------+---------+---------+