Hey,
I am creating this topic to get feedbacks of Kubernetes and LXC users on how they implemented their persistent volume storage for k8s pods.
I am currently working on a 3 nodes Kubernetes cluster inside LXC containers. At a moment came the question of persistent storage and we tried to use Rook with Ceph. It was complicated to use the block storage so we decided to create our own Ceph cluster. With that we were able to create PVC using CephFS but we wanted to use RBD instead of a File System.
When trying to use RBD instead of CephFS, it doesn’t work because Kubernetes seems to not be able to execute some commands. It is weird cause those commands can be executed manually.
When investigating, we discover that we used different Ubuntu version (the k8s nodes have Ubuntu:18.04 and Ceph nodes have Ubuntu:18.10) -> Could it be the reason of why it is not working?
Another annoying thing is was when we wanted to stop and delete one LXC container of the k8s cluster. It seems that the kernel module libceph keeps trying to connect to one of the Ceph monitors and I don’t know how to stop this kernel module apart from rebooting the host machine.
Is there anyone that managed to implement a persistent block storage solution? I am quiet stuck right now and any tips would be appreciable…!
- LXC version is
3.17 - Kernel version is
Linux n2 4.18.0-25-generic #26-Ubuntu SMP Mon Jun 24 09:32:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux - Kubernetes version is:
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.4", GitCommit:"a87e9a978f65a8303aa9467537aa59c18122cbf9", GitTreeState:"clean", BuildDate:"2019-07-08T08:51:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.7", GitCommit:"8fca2ec50a6133511b771a11559e24191b1aa2b4", GitTreeState:"clean", BuildDate:"2019-09-18T14:39:02Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}