I’m thinking the same as restricted.networks.access option but for volumes, potentially with RW/RO option as well as an option for users of restricted projects to share a volume with other projects.
Of course, I could disable features.storage.volumes and share volumes with Default… It’s all or nothing though… Default, Project A and Project B (and all other projects ever needing to share a volume) will live in a common space…
I know restricted.storage-pools.access exists, to alleviate the above, but it doesn’t work well on IncusOS where pools are not exactly the easiest to define and manage. Plus, it’s very coarse… Two projects can share the same pool but for data to be visible in both, restrictions on volumes must be removed so that they can share volumes. If restrictions on volumes are removed, any other volume contained in other pools (like local) will be made accessible too. Which is constraining in other ways.
I know buckets are a thing but not everything supports it, it’s very slow when handling a gazillion small files or doing a lot of random access, and it’s exposing files to the network, which can be restricted to a single potentially isolated IP, but it’s all or nothing again nontheless.
NFS sounds like a compelling solution… It’s better than S3 performance-wise but it’s faff to setup (requires a VM to start with and actual network work).
Ceph & co I know are the actual solution to this but it’s even more of a faff to setup.
I’d love to “just” be able to say "make local/project_a/my_data available RW (or RO) to project_b, at least on non-clustered systems but for a cluster, you want some sort of network capacity.
for me the cleanest single-host workaround is a host path, not cross-project volumes.
-mkdir -p /var/share/whatever
-on both projects set restricted.devices.disk.paths to allow that path
-attach the same source into instances in each project as a disk device
that keeps features.storage.volumes on so project-local volumes stay private. only that one host path is shared.
on IncusOS host paths are uglier. id probably spin a small nfs/smb box in a shared project and mount from both sides, or open a feature request for volume ACLs like the networks one.
Host paths are a bit clunky too for anything other than a pet system: it is out of Incus so you have to figure out of way to handle your data lifecycle separately from it (whether it’s for backups or, when time comes, when it comes to moving a volume around in a cluster)… and as you mentioned, it’s a non-feature on IncusOS.
Yes I agree for NFS/SMB/9P or similar. In practice though, kernel NFS implementation doesn’t work in LXC and neither does the main usespace implementation, NFS-Ganesha, because it uses name_to_handle_at and open_by_handle_at syscalls. Although they are nice for performance, allowing them would make container escape very easy. It looks like the ability to export custom volumes as NFS/SMB/9P should be baked into Incus itself if one does not want to use privileged containers or VMs. I’ve not tried SMB: the story there might be better.
The more I thought about it, the more it looks like the neatest approach would be an ability to export a dataset as NFS/SMB/9P natively from Incus. The only problem is that none of these are HA and the only way to make them HA is by using a distributed FS. When you do that, you might as well just use the distributed FS directly.
Considering everything, it’s likely I’ll just go volume -> virtio -> VM -> NFS -> virtio -> NetBridge after trying a bit to get a dedicated pool defined.