Struggling to understand storage

,
1

Hi

I am just getting started with Incus and want to try and avoid making mistakes where possible.

I have an existing TrueNAS server and a new mini PC which communicate together over a switch (as you would expect)

I have been reading the About storage pools, volumes and buckets - Incus documentation documentation but just want to ask a few questions.

Is it as simple as just creating a dataset on TrueNAS and mounting it to the mini PC via NFS? Then installing Incus and pointing it at that mount point as a storage location?

This would use the zfs driver as I understand it.

I would like to add two more mini PCs in the future to create a cluster. Will each of these need its own dedicated dataset on the TrueNAS server? Would this support live migration?

Or should I use TrueNAS - truenas - Incus documentation instead? This looks a bit ‘new’ though.

The goal is to have a three node cluster with live migration and HA. Would either method work for this aim?

Thanks

2

If exporting storage over NFS, your only option would be to use the dir backend on top of that, this won’t be particularly pleasant due to the lack of copy-on-write or snapshot support.

You could export over iSCSI instead which would give you a block device that you could then use with most of the storage drivers (ZFS, btrfs or LVM) but that won’t get you much control on the TrueNAS side as you’d just see one big volume on it.

Note that those two options only really work with a standalone Incus deployment.
When clustering and wanting live-migration/HA, you want proper concurrent access from multiple servers, which neither of those options really get you.

That’s where the new TrueNAS storage driver comes in as that uses TrueNAS to create individual volumes for each container/VM and then export that over iSCSI to the server running Incus. Giving you something similar to using iSCSI+ZFS but with more granularity on the TrueNAS side and with the option to eventually cluster Incus and keep using the same storage.

An alternative to that would be to use TrueNAS with iSCSI, export the same volume to all systems, then use clustered LVM on top of that to give you a clustered layer, but that’s quite a bit of extra complexity when the native TrueNAS driver now exists and avoids all that.

1 Like
3

Thank you for your reply.

I tried the TrueNAS storage driver as it seemed the best fit for me but for whatever reason I got stuck on the activate step.

My other iSCSI volumes have CHAP set on them and although the new volume should not have been asking for this it seemed to be for whatever reason.

In the end I went with the second option of an iSCSI disk which I presented to my host in the traditional way. This seems to work fine as a first step while I learn. I will revisit the TrueNAS driver later.

4

Can you report the CHAP issue in this repo: GitHub - truenas/truenas_incus_ctl: TrueNAS CLI Admin tool for Incus and Consumers

That way the TrueNAS folks working on the Incus integration can get this resolved.

1 Like