I'm using OVN in the public cloud with AWS and Oracle. Change my mind

So I did a thing. Basically I just was following the OVN docs and seeing how I would apply it in AWS. I happened to get it working by creating a bridge network on each node and allowing ip forwarding. The ip range I used was just a random ip range that I am not using for my VPC range. Everything seemed to work fine in the end. Containers could talk to each other and ACL seemed to work fine. Even peering connections worked just great. Communication between containers on different nodes worked too. After doing this I found that using OVN in the public cloud is not recommend or shouldn’t work here https://linuxcontainers.org/incus/docs/main/explanation/networks/. Though everything seems to be working fine. The only thing I have not been able to verify works with this process is the load balancers. However you can get around that with proxy devices to expose services.

I documented what I did in this project. https://github.com/Xachman/incus-aws

My question is if this is working fine is there any reason I shouldn’t continue to use this process?

I have this working now in oracle cloud as well. And it seems to be great!

I’ve seen a number of people do a similar set up, basically using incusbr0 as the uplink network for OVN.

This does work just fine, the main downside compared to the recommended approach of having a physical shared uplink network accessible to all servers (which isn’t possible in the cloud) is that each OVN network will pick one of the server for its ingress/egress with the outside world (you can see that in incus network info), that server will then NAT your traffic to its own IP address.

Again, this is usually fine, the only downside is that if that server was to die or be brought offline, now OVN will move the ingress/egress through another server, which will then perform its own NAT, resulting in all established connections to be broken and need re-establishing.

That’s usually not a concern for most folks doing this as they don’t need that level of redundancy, but just something to keep in mind when performing maintenance as it means that evacuating and restarting one server may affect the network connectivity of a bunch of instances running on the other servers.

Thanks for replying! I was able to confirm that this certainly is the case. This isn’t a big deal for us so I am willing to live with it.

Thanks for all your hard work! We are moving our project over from LXD. Before we were using fan overlay but that has fallen out of favor. So the most obvious solution seemed to be OVN.

Is there any other solution incus provides that would work well for public cloud networking?

Not currently. I have some ideas for a basic overlay we should be able to do with our built-in BGP support and that will come needed soon enough with the ongoing Incus OS work, but it’s not there yet.

It also won’t be nearly as featureful as OVN, so no firewalling stuff, no load-balancers, no forwards, … just something that allows a bunch of instances to communicate with each other and with the various hosts as well as being able to move those instances without them having to change addresses.

1 Like

Ok! As far as I can tell OVN works well enough in the public cloud. All my testing shows its functional so we are going to move our app over to incus sometime this month.

Thanks for all your help!