Hey,
I got an email since you referenced my post, so I figured I’d jump in here.
I’m going to misuse a few words here but that’s because I want to phrase my response in a certain way.
IncusOS only runs the “client” OVN services. You need to set up the “server” OVN services. There are two ways to do this, I have done both and there isn’t very many pros/cons to either one.
Option 1: OVN “servers” in LXCs
Essentially what you want to do here is make sure each of your IncusOS servers has two NICs. I wasn’t able to get it to work with just one NIC.
What I did is I set up a “bridge network”, called incusbr0 and a “physical network”. The reason you need both is that one of them exposes your LXCs directly to the “outside” and the other one gives each host an IP and performs NAT.
What I did is in my “default” project, I created an LXC for each IncusOS host and pinned it to that host, that way if a node goes down, I dont risk losing my networking with it. (split-brain databases are manageable, but losing your database entirely is not so manageable).
For each LXC in the default project, I named them ovn-central-2, ovn-central-3, ovn-central-4, but that doesn’t really matter. You need to assign each one of those LXCs to the PHYSICAL network. This allow for them to use your home router’s DHCP reservations/routing. The physical network is like a Layer 2 switch.
On those LXCs, you can follow any normal tutorial on installing the “server” part of OVN, which is only the databases. For Debian, the package is just ovn-central.
Then when you create a new Project, inside of that project, you create a new OVN network. This OVN network will uplink to the Bridge network, which provides NAT and OVN DHCP. From here, you should be able to hop back into the IncusOS Documentation and finish up your network.
Option 2: Dedicated OVN “servers”
The process is largely the same and this is what I am running right now. I got 3 IncusOS hosts and I have 3 mini PCs. The Mini PCs are running debian with ovn-central. and I use Cockpit and Cockpit-pcp to remotely manage them.
This setup requires you to have some spare endpoints laying around, but your IncusOS hosts dont need the Physical network, they only need the Bridge network.
I hope that helped fill in some gaps. I actually broke my entire home network today and I think I need to rebuilt my servers from scratch, so if I end up doing that, I can write a detailed blog post about each step in rebuilding the cluster with multi-tenant OVN. If that happens, Ill let you know.
