I have been playing with ceph+LXD for the past couple of days and wanted to build a proof of concept on a VPS to showcase something with a colleague.
I installed a CEPH instance on a VPS .
the instance is up and is only complaining about the lack of OSD nodes (not ideal but this is just a quick POC; a single node would do)
Then i tried installing LXD on another VPS only for lxd init to hang
and lxc storage info remote to give me this error:
Error: Failed to run: ceph --name client. --cluster ceph df -f json: 2022-08-11T15:14:26.623+0000 7ff494a95700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
[errno 13] RADOS permission denied (error connecting to the cluster)
Do note that just running the command ceph status on the lxd machine connects the ceph node without any issues
I feel like LXD is using a different CEPH client from the one installed system wide but that’s just a guess.
any help would be appreciated!
Thanks.
setup information:
LXD version is 5.4 (snap)
ceph-common is 17.2.3 (from apt)
this also might be relevant
root@lxd-s-2vcpu-2gb-fra1-01:~# snap get lxd ceph
Key Value
ceph.builtin false
ceph.external true
and here is the output of what i was trying to give to lxd init
The snap package includes the ceph client, it doesn’t use your system’s ceph by default.
You can change that with snap set lxd ceph.external=true followed with a systemctl reload snap.lxd.daemon
Ok this fixed the first issue now lxd can connect to ceph no problems.
But now I get this error
Error: Failed to create storage pool "default": Failed to run: ceph --name client.admin --cluster ceph osd pool create lxd 32: Traceback (most recent call last):
File "/usr/bin/ceph", line 1326, in <module>
retval = main()
File "/usr/bin/ceph", line 1246, in main
sigdict = parse_json_funcsigs(outbuf.decode('utf-8'), 'cli')
File "/snap/lxd/current/lib/python3/dist-packages/ceph_argparse.py", line 847, in parse_json_funcsigs
cmd['sig'] = parse_funcsig(cmd['sig'])
File "/snap/lxd/current/lib/python3/dist-packages/ceph_argparse.py", line 793, in parse_funcsig
newsig.append(argdesc(t,
File "/snap/lxd/current/lib/python3/dist-packages/ceph_argparse.py", line 673, in __init__
self.instance = self.t(**self.typeargs)
TypeError: __init__() got an unexpected keyword argument 'positional'
I do realize this is technically a ceph issue.
but running that same command creates a new pool no problems.
Having the same problem again and setting ceph.external=true isn’t helping
Environment
ceph 17.2.3
lxd edge (git-ae54a13) AND lxd stable (5.44)
lxd.ceph.external=true
Issue
root@nyc3-lxd:~# lxc --debug storage info remote
...
"storage": "ceph",
"storage_version": "17.2.3",
"storage_supported_drivers": [
{
"Name": "ceph",
"Version": "17.2.3",
"Remote": true
},
{
"Name": "cephfs",
"Version": "17.2.3",
"Remote": true
},
{
"Name": "cephobject",
"Version": "17.2.3",
"Remote": true
},
...
]
}
}
DEBUG [2022-08-18T13:07:09Z] Sending request to LXD etag= method=GET url="http://unix.socket/1.0/storage-pools/remote"
DEBUG [2022-08-18T13:07:09Z] Got response struct from LXD
DEBUG [2022-08-18T13:07:09Z]
{
"config": {
"source": "lxd-ny"
},
"description": "",
"name": "remote",
"driver": "ceph",
"used_by": null,
"status": "Pending",
"locations": [
"none"
]
}
DEBUG [2022-08-18T13:07:09Z] Sending request to LXD etag= method=GET url="http://unix.socket/1.0/storage-pools/remote/resources"
Error: Failed to run: ceph --name client. --cluster df -f json: exit status 13 (2022-08-18T13:07:09.528+0000 7f978effd700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
[errno 13] RADOS permission denied (error connecting to the cluster))
And all LXD ceph operations fail.
I am unable to create a new storage pool or list existing PENDING ceph ones.
So what gives?
I tried purging lxd and installing it again and that didn’t work
Also on a side note that might help narrow down this issue:
The ceph client I have installed supports reading from /etc/ceph/ceph.client.admin.keyring
But I get this error message when trying to do a ceph operation from lxd
root@nyc3-lxd:~# lxc storage info remote
Error: Failed to run: ceph --name client. --cluster df -f json: exit status 13 (2022-08-18T13:13:59.796+0000 7f74c749a700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client..keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
which is fixed by copying /etc/ceph/ceph.client.admin.keyring to /etc/ceph/ceph.keyring