Setting up containers to run on a minIO distributed setup?

LXD
1

Does anyone have any experience setting up the lxd storage backend on a minIO distributed setup?

I am looking to setup something thats easier to maintain than Ceph and i found MinIO which is a really nice distributed object storage solution. The nice thing about minIO is that it works on any type of file system (ext4,xfs,etc.).

My initial thoughts were to create a distributed object storage (because i am looking for very fast I/O performance), format it and then use it as the storage backend for my LXD installation, I could do the format as LVM and then just use all of my data available , but for some reason when i am asked to provide the path to the block device it simply doesn’t want to work. Is there something I am doing wrong ? maybe i should use btrfs ? or zfs (although i try to stay away from zfs)?

Right now i have 3 nodes that each carry 8 Micron 7300 MAX U.2 drives . Each node has a Mellanox ConnectX6-DX card (2x100Gb qfsp+) and they are connected to a Mellanox SN2700 Switch (32 x 100Gb ports).

Again the reason I am staying away from ceph is just for simplicity. I may have to switch over to it from what i have been reading (and also because i just cant find much on using MinIO with LXD, although it sells itself as the perfect Object Storage solution for containers).

Reference:

2

Are you looking to run this inside an LXD container or VM?

Inside a container, you will likely find that the if the app you are using tries to create partitions/filesystems ontop of the block device inside the container that this won’t be allowed.

You can pass in a directory as a disk that can then be used as storage directly using the disk device, see https://linuxcontainers.org/lxd/docs/master/instances#type-disk