Incus as the hypervisor for a single 2U server in colocation

Incus
1

Hey folks!

I’m trying to figure out if I can leverage Incus for a single baremetal server in Colocation. For some silly reasons, it’s running raw ESXI + Vcenter on top, all in a magical 2U.

Some specifications

  • R720XD
  • Inbound internet through a /28 link.
  • Need to do BGP with a public ASN and an ARIN assigned /24.
  • Actual workloads would be a mix of VM or LXC.
  • VM would be assigned to an internal network + sometimes to the /24 public space or just the /24.

From the Incus perspective

Networking

  1. I would use the BGP integration to advertise the public /24
  2. I would use the Bridge setup and setup two networks (private + public), VMs would be attached to either/or.
  3. The bridge setup would provide either NAT or direct BGP internet access

Storage

  1. All storage is local to the server / spinning drives.
  2. Pools would be split at the hardware level + partioned at the OS level
  3. I would favor BTRFS partitions as DIR seems simpler, but lacks all the features

A quick diagram of what it would look like

image
image1815×1184 139 KB

It seems to make sense, but not 100% sure. Looking to avoid some nice gotchas before I wipe everything.

Thanks!

1 Like
2

Hey there,

I wouldn’t use Incus’ BGP to directly advertise your /24 to your provider as Incus doesn’t let you configure what size prefix to advertise, instead it’s really meant to advertise to top of rack switch or router which can then advertise a larger prefix to the external peers.

In your case, as you’re dealing with a single server and no OVN in the mix, your best bet is to just use the host server as the router. Install FRR on the host, configure that to peer with your transit to advertise your /24 and get your routing table (given single peer I’m assuming you’re just getting a single default route).

You’ll then effectively have your /24 routed to your host and can further route it to your Incus bridge.

I’d do things a bit differently if OVN or some kind of redundant hardware was involved, but for a single server system, that’d be the way to go.

1 Like
3

Ah, I missed that BGP Incus part. I currently have Pfsense doing the FRR part, but I was looking at collapsing all of it as much as possible. It is single homed with a default route but I was looking at full route (just for fun!)

Having it on baremetal works as well.

Thanks for the insights!

4

At least you’re going to get great througput from the Linux Containers infra to your server seeing we’re likely in the same building or the one down the road :wink:

(Given the single 2U server, you’re likely at QC4 whereas with our larger footprint we’re at QC1)

5

Haha, you got me, I will never have downloaded packages this quick! :grin:

It’s by reading your 3 node HA blog post that I dug a bit deeper + the rock solid scaled infra from NorthSec that I found out about Incus.

I’ll give this a go, thanks a lot for the feedback!