Incus multi-tenant dream scenario

1

I am but a common business man with a big vision. Started on proxmox nearly 2 years ago. Got addicted to launching VM’s for every consumer-grade open source app being released on github daily. Self hosting biz websites… then replacing thousands a month in SaaS subscriptions with their open source alternatives… yea, I could have saved six figures had I dived into this shit earlier.

Now, If you’ve spent the amount of time I have in the VM/LXC/Docker world, you know why Incus kicks ass. You’re here so I don’t need to explain that part.

But I need something explained to me if someone would be so kind:

How exactly would you establish a declarative cloud-native (database-less!) multi-tenant container network system on a single VM?

I’ve tried the following:

  1. The very handy unix-account based project creation method standard in Incus.

Problem: lots of port forwarding and proxy rules, messy for multiple users running same web apps on single host.

  1. Host-level reverse proxy with DNS resolver per incus bridge.

Problem: Still deal with conflicts using same-name containers across projects.

  1. (dream) - VRF isolation at the host level, connecting project-based bridges to dedicated VRF tables.

These forums are pretty much all technical, so let me just cut to the chase of why number 3 or any solution to achieve the same thing would be magical for many of us from a biz use case perspective:

Reason 1 - If every incus project was able to share the same subnet and ports (which is the point of VRF), you can very quickly reach a cloud model that provides a new-user signup flow provisioning an entire virtual private cloud WITH live and production ready web apps in it… 100+ times on a single server… in about a second per user or more.

Reason 2 - Reason 1 basically means we could ditch docker with a more performant container system, and essentially standardize a “logical VM” network where the shell environment inside every container is nearly identical to a full blown VM - but with those full blown VM’s being portable and tiny like k8’s or docker swarm.

Reason 3 - With 1 and 2 in mind, you now don’t need a database for Incus - which currently runs on sqlite… Which is great for some private cloud fun, but instantly limiting when it comes to what REALLY needs to be replaced in the cloud market… which, is in fact, the entire cloud market.

So, provide that solution, and I’ll show you how to replace a hyperscaler of the AWS/GCP tier starting with a single VM host.

I’ve invested a lot of time and money into solving this problem and have high hopes… but I’m officially hitting a wall with how best to actually provide this solution I envision. Any takers?

1 Like
2

Did you play with OVN?

OVN can be used on a single system, it’s not only for clusters.

Once you have OVN setup, each project can own its own set of networks, including using the exact same subnets as are used in other projects (proper multi-tenancy).

You can then grant some “external” addresses to each project which can either be routed directly to instances or used for network forwards or load-balancers.

3

Yes, I’ve studied every single option in the documentation for networking. Have watched your YouTube videos on OVN as well. The reason I’m so hung up on VRF-based approach is because (if) its possible, it’s a lot better. The industries I focus on are often ones where small gains in efficiency/performance define your profit margin.

Utilizing kernel-level networking stack (iptables2 with vrf seemes to be standard on all minimal linux distros even alpine) means in theory, all I should need is a bash script (or cloud init config) running alongside a fresh minimal VM with nothing but Incus installed on it to deliver the desired result.

Everything I’ve read and every AI model I’ve hashed this out with agrees. What’s so annoying is not knowing if there is something dumb simple I’m missing from your perspective that would create the final working recipe. I know I’ve done the hard part already… so it has to either be a stupid fix or non-obvious blocker at this point. And I understand this type of stuff probably gets to the enterprise/paid level of work in most cases… but if you or any of the other contributors would be willing to link up on solving this, I can share a lot of notes and point out exactly where each VRF based method seems to get hung up.

My earlier post shares some notes on that end, so yea, again… if anything comes to mind as a clear blocker at least I could run with that. Or is VRF stuff just not a particular core strength/focus in your eyes?

4

Aside from my remaining stubborn desire to take the VRF approach… I suppose the single-server OVN network method might have been overlooked. From my understanding, OVN isn’t baked into the incus binary and requires additional packages. But if its in-fact dynamic enough to handle projects coming and going from VM’s regularly with a similar scriptable approach to what I’ve been working through with VRF’s, then I’ll start spending time there instead for now.

(letting this sit for a minute before marking as solved in case you or others have other input regarding VRF method)

5

Aha… I’m reviewing the 6.12 release notes right now and think some of this ACL and SNAT stuff is probably related to my needs. So that’s promising as well. I’ll try to give this a final college try utilizing the latest features and report back.