Connection to a Ceph cluster

Naah. I give up :disappointed:

I deployed a 3-node Ceph cluster. It works fine.

[root@ceph01 ~]# ceph -s
    id:     47fb8a70-6bfc-425d-9934-a2b77155d36c
    health: HEALTH_OK

    mon: 1 daemons, quorum ceph01
    mgr: ceph02(active), standbys: ceph01
    osd: 6 osds: 6 up, 6 in

    pools:   1 pools, 1 pgs
    objects: 0  objects, 0 B
    usage:   7.2 GiB used, 41 GiB / 48 GiB avail
    pgs:     1 active+clean

Then I tried to create a storage pool from an LXC server.

lxc storage create pool1 ceph ceph.cluster\_name=ceph

But I constantly get the same error.

**Error: Failed to create storage pool 'ceph': Failed to run: ceph --name client.admin --cluster ceph osd pool create lxd 32: Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)**

Depite the fact my LXD server can ping a Ceph cluster node (one of 3 nodes).

[root@node4 ~]# ping ceph
PING ceph ( 56(84) bytes of data.
64 bytes from ceph ( icmp_seq=1 ttl=64 time=0.284 ms
64 bytes from ceph ( icmp_seq=2 ttl=64 time=0.339 ms
64 bytes from ceph ( icmp_seq=3 ttl=64 time=0.490 ms
64 bytes from ceph ( icmp_seq=4 ttl=64 time=0.314 ms

What am I doing wrong?

Can you show your /etc/ceph/ceph.conf?

LXD ships an older version of Ceph within the snap currently so some options in there may confuse it.

Yeah, I can. This is the node I’m trying to connect.

fsid = 47fb8a70-6bfc-425d-9934-a2b77155d36c
mon_initial_members = ceph01
mon_host =
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public network =

Is there any mistake? Cannot quite understand what’s the problem.

Hmm, no, that looks correct.
Can you show:

  • nsenter --mount=/run/snapd/ns/lxd.mnt ls -lh /etc/ceph/

Nah…none of the server succeded

Hmm, what LXD version is that?

LXD is started via snapd manager.

Ah, I’m running CentOS 8.2 btw.

Ah that may explain some of this :slight_smile:


nsenter --mount=/run/snapd/ns/lxd.mnt /bin/ls -lh /etc/ceph

Got smth.

Got another progress.


nsenter --mount=/run/snapd/ns/lxd.mnt /bin/ls -lh /etc/ceph/


Oh, okay so it may be a very simple problem :slight_smile:

Do you have a /etc/ceph/ceph.conf and /etc/ceph/ceph.client.admin.keyring on every one of your LXD nodes? If not, then that’s the issue, you need those two files in there so LXD can access Ceph.

You can simply copy those two from one of your Ceph cluster nodes.

Ahh! Thanks a lot! :slightly_smiling_face:

What was the magic?

In general to be able to connect to ceph, you need the machine connecting to it to also have the ceph.conf and a keyring. I usually go as far as installing ceph-common so I can run the ceph status and related commands on them too.