HowTo: CephFS with LXD Clustering

LXD
1

Instructions weren’t quite clear on how to set this up and kind of misleading, so here’s how you do it.

Make sure you have at least 1 MDS configured and online.

Creating the CephFS:

ceph osd pool create cephfs_lxd_data 64
ceph osd pool create cephfs_lxd_metadata 64
ceph fs new lxd_cephfs cephfs_lxd_metadata cephfs_lxd_data

Creating the storage pool in LXD:

lxc storage create remotefs cephfs source=lxd_cephfs --target {lxd_hostname}

Repeat above for every host in your cluster. replace {lxd_hostname} with actual hostnames.

lxc storage create remotefs cephfs

Now you should get a storage pool created message in the output.

Some comments about the 3.15 release notes, this line:

lxc storage create test cephfs source=persist-cephfs/castiana

and this error message

Error: Config key 'source' is node-specific

Lead users to believe that they need to specify their hostname in the source line as well but if you do that you get:

cephfs.path must match the source

Which is not immediately clear what you should be doing, lead me down another rabbit hole trying to configure cephfs.path which was obviously not the right thing to do.

Hopefully, this is helpful to someone else.

2 Likes
2

Ah yeah, indeed my Ceph FS demo in the announcement wasn’t in a clustered environment but instead having my laptop use it.

3

I got stuck on the first command:

ceph osd pool create cephfs_lxd_data 64

2019-08-30 11:03:07.938823 7f5c4c0be700 -1 Errors while parsing config file!
2019-08-30 11:03:07.938824 7f5c4c0be700 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2019-08-30 11:03:07.938825 7f5c4c0be700 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2019-08-30 11:03:07.938826 7f5c4c0be700 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)
4

Awesome, thanks for the great notes!

With CephFS, are you able to move containers to other servers without out copying/cloning them first? Does a simple “lxd move” command work pretty fast?

5

Containers can’t be on CephFS, thye can only be on Ceph (RBD), CephFS is only used for custom storage volumes.

In any case, yes, using Ceph does allow for cloning/copying/moving within a cluster without actually needing the source server send any bits to the destination. You can also move containers from a server which is offline without any problem.

6

Thanks Stephane. From your comment, I seems Ceph (RDB) can be used to create a real Highly Available LXD environment since the containers live on a shared filesystem mounted to multiple LXD servers at the same time. Is this correct?

7

Not mounted at the same time (this would cause corruption) but certainly available to multiple servers, yes. If a server goes down, lxc move blah --target some-server will let you move the container to a server that is online and let you bring the container back online.

cephfs also allows you to have multiple containers on different servers seeing the same data for application storage. This can be useful for websites or other shared on-disk content, allowing for building multi-server applications and making things more resilient.

8

Thanks Stephane. Appreciate the info!