IncusOS with secondary storage configuration to host NVMe/TCP

IncusOS
, ,
1

I have 4 mini PCs with a 256gb SATA drive and 1tb NVMe and 2*2.5g ethernet ports each.

What I’d like to do is install IncusOS the 250gb SATA drives, forming a cluster. Then configure a NVMe over TCP storage device on those same devices to use as a network shared storage pool.

The documentation I’ve read on storage devices on IncusOS doesn’t seem to indicate this is possible. Has anyone managed to configure anything similar to this, or is there an alternative I should use? (I have had success with Debian running Incus and hosting an iSCSI lvm-based storage pool, but am looking to start over with IncusOS and NVMe/TCP if possible)

2

IncusOS can easily connect to remote iSCSI or NVME storage (see services).

But here you’re mentioning that each server has a local 1TiB NVME drive.

I’m not sure what you intend to do with that as even if IncusOS allowed exporting it over the network, you still wouldn’t really have something that’s usable as cluster-wide storage.

For what’s effectively a hyperconverged cluster, you’d typically want to use something like Ceph or Linstor to consume that local NVMe drive and turn it into a cluster-wide storage pool instead. IncusOS has support for running as a Linstor Satellite which allows you to do that with Linstor and I’m currently working on adding Ceph as well by having the Ceph services run in containers rather than directly on IncusOS.

1 Like
3

Thanks, I’ll look a bit more into LINSTORE, that might be the answer I’m looking for.

I know my design is a bit odd, doing double duty of container hosting and distributed storage management, but its the redundancy I’m going for. If I had a separate device for network storage it would become a single point of failure, so I would need two to eliminate that. With what I’m doing, each device hosts everything so I can lose one and still maintain a quorum on both incus cluster and storage.

Thanks for your time, if I get it working I’ll come back and post here (probably a couple weeks for me to find the time to do it).

4

Yeah, so you’ll definitely need Linstor or Ceph to do what you want as they are the only two solutions that let you create a consistent storage pool using local storage over multiple servers.

1 Like