Need some guidance to setup a storage pool with CEPH

LXD
1

There are bits and pieces describing commands to create a CEPH storage pool.

However, I would be grateful to see a few example commands. I have a working LXD cluster (currently with LVM storage). I have also setup a CEPH cluster on the same nodes of the LXD cluster. The CEPH status shows

root@luts:~# ceph --status
  cluster:
    id:     e5b01928-f9b0-11ea-a593-3f5e8a462567
    health: HEALTH_OK
 
  services:
    mon: 5 daemons, quorum luts,angstel,maas,roer,rijn (age 4d)
    mgr: luts.ohbhlv(active, since 4d), standbys: angstel.suocfa
    osd: 5 osds: 5 up (since 115m), 5 in (since 115m)
 
  data:
    pools:   1 pools, 1 pgs
    objects: 4 objects, 0 B
    usage:   5.0 GiB used, 9.1 TiB / 9.1 TiB avail
    pgs:     1 active+clean

Then I tried this

root@luts:~# lxc storage create cephpool ceph
Error: Pool not pending on any node (use --target <node> first)
root@luts:~# lxc storage create cephpool ceph --target luts
Storage pool cephpool pending on member luts

Meanwhile, the storage is “PENDING”. I don’t know where to look for detailed status.

Somewhere I read that I have to copy /etc/ceph/ceph.conf and /etc/ceph/ceph.client.admin.keyring to the other LXD cluster nodes. These details seem important to me, but I haven’t seen a how-to or tutorial describe this. And there could be more important details.

In other words, a How To, or Tutorial would be appreciated.

2

As with any storage pool in a LXD cluster, you must first configure the node-specific config with:

lxc storage create NAME TYPE source=SOURCE --target TARGET

Once that’s done on all nodes, you may create the global record with:

lxc storage create NAME TYPE

So in this case, you’re looking at something like:

  • lxc storage create cephpool ceph source=my-osd-pool --target luts
  • lxc storage create cephpool ceph
3

OK I see.

LXD is doing magic stuff when it comes down to interpreting source=. Going through the source of driver_ceph.go I see that LXD will create the “osd pool”.

root@luts:~# lxc storage create cephpool ceph source=lxd-osd-pool --target luts
Storage pool cephpool pending on member luts
root@luts:~# lxc storage create cephpool ceph source=lxd-osd-pool --target roer
Storage pool cephpool pending on member roer
root@luts:~# lxc storage create cephpool ceph source=lxd-osd-pool --target maas
Storage pool cephpool pending on member maas
root@luts:~# lxc storage create cephpool ceph source=lxd-osd-pool --target rijn
Storage pool cephpool pending on member rijn
root@luts:~# lxc storage create cephpool ceph source=lxd-osd-pool --target angstel
Storage pool cephpool pending on member angstel
root@luts:~# lxc storage create cephpool ceph source=lxd-osd-pool --target ijssel
Storage pool cephpool pending on member ijssel

So far so good

Next,

root@luts:~# lxc storage create cephpool ceph
Error: Failed to run: ceph --name client.admin --cluster ceph osd pool create lxd-osd-pool 32: unable to parse addrs in '[v2:172.16.16.45:3300/0,v1:172.16.16.45:6789/0]'
[errno 22] error connecting to the cluster

Is that the problem that I need some files in /etc/ceph on each node? (ceph command is part of the lxd snap, right?)

4

You may be running a Ceph 15 cluster with LXD bundling an older 12 client.

If that’s the case, on all nodes, run:

  • snap set lxd ceph.external=true
  • systemctl reload snap.lxd.daemon

This will tweak the snap so it uses you system’s ceph client rather than the older built-in one.

5

And I need to install ceph-common on all LXD nodes, right?

6

That’s correct, yes.

I usually make sure that ceph status works on all nodes before having LXD interact with it.

7

Yippee, that worked

root@luts:~# lxc storage create cephpool ceph
Storage pool cephpool created
root@luts:~# lxc storage list
+----------+-------------+--------+---------+---------+
|   NAME   | DESCRIPTION | DRIVER |  STATE  | USED BY |
+----------+-------------+--------+---------+---------+
| cephpool |             | ceph   | CREATED | 0       |
+----------+-------------+--------+---------+---------+
| local    |             | lvm    | CREATED | 28      |
+----------+-------------+--------+---------+---------+
8

I tried running ceph status on all my nodes and all of them display the following error:

root@n1:~# ceph status
Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')

Any idea what could have caused this? I have also followed all the steps described above.

9

You need to get a copy /etc/ceph/ceph.conf and /etc/ceph/ceph.client.admin.keyring from the first ceph node. If there is a MON daemon present you can modify ceph.conf to use its own IP address (there one MON listens to).

10

Seems I have to research more regarding Ceph. I have installed ceph-common on all nodes, but none of them contain de ceph.conf file, the only file present in /etc/ceph/ is a rbdmap file. Do I need to create a Ceph cluster first before proceeding to adding it to my LXD cluster?

11

Yes, that is what I did, create a CEPH cluster first. I don’t think that LXD can setup the CEPH cluster behind the scenes. All it can do is to create a pool on an existing cluster.

I’m saying all this as a complete ceph newbie, so take it with a grain of salt :slight_smile:

12

Ok thank you for the help, I will try creating the Ceph cluster first.

13

BTW. My intention of this topic was that someone could point me to a how-to or a tutorial. But I’m afraid there isn’t any.

So, whatever we write here might help others who want to investigate LXD+CEPH.

1 Like
14

That is a good train of though, a more centralized procedure documentation would be helpful for all.

Regarding your cluster setup, are your OSD nodes also your MON nodes?

15

Take a look at [1], where Stéphane suggested the following:

The way I do it is I make sure that I’m dealing with at least 3 systems (otherwise consensus for both LXD and Ceph will be a problem), then pick 3 to host both OSD and Ceph services (MON, MGR, MDS) with the others just running OSD on their disks.

And

You need to account for the extra memory and CPU usage that comes from running those services, but it’s usually okay to do it that way, eliminates the need for dedicated storage hardware and lets you spread your storage across more systems.

Althoung I’m a big proponent of using standard Ubuntu/Debian packages I ended up to install with this in /etc/apt/sources.list.d/ceph.list. In hindsight it is probably not necessary.

deb https://download.ceph.com/debian-octopus focal main

First I installed docker.io because otherwise cephadm will complain or install it on its own.

apt install docker.io
systemctl enable docker
systemctl start docker

Then I started the ceph cluster with

mkdir -p /etc/ceph
cephadm bootstrap --mon-ip 172.16.16.45 --ssl-dashboard-port 9443

Without specifying the dashboard port there is a conflict with 8443 already in use by LXD.

Next, install ceph-common on the next ceph node. After that, on the first ceph node you can add the new node as follows (where second-node is the hostname of the new node.

ceph orch host add second-node

And so on for the other ceph nodes

Finally, add an OSD for each disk (/dev/sdc in this example)

ceph orch daemon add osd second-node:/dev/sdc

[1] Is there any advise to deploy a ceph node on a LXD host

16

Thank you for the guidelines! I will try and run them on my VMs.