Hello. I have 4 LXD nodes with ceph’s RADOS Block Device and nodes are in one cluster. I use ceph as a storage backend for my LXC containers. So, can I use one volume within multiple LXD nodes?
The same question has already been discussed here
Assuming you’re talking about attaching custom volumes to containers, I’m not sure that LXD enforces that right now, but you should NEVER attach a custom volume to two containers that are on two different hosts.
CEPH doesn’t have any protection against this and doing so will lead to data corruption in the filesystem used for that custom volume.
Doing the same when containers are on the same host is perfectly fine as the Linux kernel will effectively treat it as a single mount, avoiding the data corruption issue which results from multiple concurrent mount of the block device.
But ceph allows you to mount one rbd volume to multiple hosts. And I think that people usually use this filesystem to have an ability to run their apps on multiple hosts while app’s data is in a ceph volume. You can mount one ceph rbd device to multiple docker containers and tbere is an official container developed by ceph’s developers which allows you to attach a volume to docker container.
I think that the best way to solve this problem is to mount Ceph FS(not RBD) to /var/lib/lxd
cephfs effectively being a networked filesystem should be fine as far as multiple parallel mounts is concerned. rbd most definitely isn’t, unless it’s used with a clustered filesystem on top (something like gfs2).
It’s really quite simple, unless the filesystem used on top of the block device is aware that it may be mounted in multiple places at the same time and has some built-in support for distributed locking and journaling (as something like gfs2 does), it will badly corrupt itself when multiple systems attempt to write to it.