Using the latest lxd snap I’m seeing some oddities with Ceph snapshot handling as well as the inability to use Mimic’s balancer:
storage_version: ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
trystan@sionis:~$ sudo ceph osd set-require-min-compat-client luminous
Error EPERM: cannot set require_min_compat_client to luminous: 4 connected client(s) look like jewel (missing 0x800000000000000); add --yes-i-really-mean-it to do it anyway
There have been significant changes to RBD image handling in Luminous as well as Mimic and I’m curious if an option could be added to set a client version.
If I were to migrate to a ‘normal’ apt based LXD on 3.0 would I be able to utilize the Mimic client packages?
The snap bundles the CEPH tools from Ubuntu 16.04, so that’s indeed likely to be the issue here.
I don’t know how backward compatible CEPH is, if we expect the latest tools to behave fine against older servers, we may be able to update the snap to bundle a more recent version.
As for the deb question, yes, the deb just uses whatever is available on the system.
With Cosmic’s move to a ‘transitional’ snap install when installing the lxd deb package, there will be no way to utilize a newer version of Ceph for those of us with Luminous/Mimic cluster features.
Also considering that Ceph is the only available ‘remote’ storage option this to me is a significant problem.
An option to use the packaged distribution’s Ceph binaries or decoupling Ceph from inside the snap might be worth considering.
We do intend to switch the LXD snap onto an 18.04 base which will get us CEPH 12.2.8.
It may also be possible for us to effectively build the CEPH tools from source, making it easier to switch to whatever the best upstream release is but that may be more work.
The Ceph team has recently increased the release of LTS branches to a more rapid model so creating a pipeline to build from source might be worth a look, especially if there’s no way to use an external dependency in the snap.
I started looking into this yesterday, unfortunately building the two CEPH commands we need (ceph and rbd) from source seems a lot more painful than expected…
Unless I find an easy way to do this, the obvious next step will be to switch the snap to an Ubuntu 18.04 base which would give us 12.2.8.