Anyone using Incus cluster with POSIX compliant distributed file systems (e.g. like Weka, VAST, Lustre etc.). We have and use them in our Slurm HPC cluster, but I was wondering about usage as an Incus storage backend (dir storage driver, as hosts just sees these as normal POSIX FS) for VMs and containers.
Any insights about this @stgraber?
Often even when POSIX compliant, locking may be slow/tricky, so running a full container from it may be “interesting”. They’re also not terribly likely to support some of the more advanced filesystem features like VFS idmap, leading to sub-optimal handling of file ownership.
That’s why we currently limit cephfs to extra storage volumes and are looking at doing the same for the upcoming nfs driver.
But technically nothing prevents you from mounting one of those filesystems and running with the dir storage driver and see whether any of those limitations are an actual problem for your environment.