BGP-to-the-host on IncusOS with unnumbered works, how do routes get back in?

IncusOS
,
1

Hi all, new to the forum!

We’re building a platform with Incus/IncusOS and the network design is a very plain BGP to the host on a pure IPv6 leaf–spine fabric, no overlays, no VRFs. The hosts announce what runs on them, bare-metal tenants announce their own prefixes, IPv4 is only present as v4-over-v6 (per RFC 8950). Two links per host towards two leaf switches for the main traffic, unnumbered on both, private ASN per device and identical config on every host.

Recently the new bgp.peers.NAME.interface added a working announce side for BGP unnumbered. Works out of the box, extended next-hop included. Thanks!

Seeing this I thought I should ask, the other direction is the next problem for our design. The incus BGP speaker installs nothing of what it receives, and there’s no FRR on IncusOS, so the host can’t learn its default from the leaves. ::/0 we can get from RA as IncusOS accepts RAs on interfaces without static addresses, so for v6 only this is fine. But 0.0.0.0/0 has no RA equivalent; it needs 0.0.0.0/0 via fe80::1 dev link0 per uplink (our leaves present a fixed fe80::1), which networkd can do but the IncusOS route schema can’t express … yet?

So from that discussion, a small wishlist:

routes[].dev so a link-local next hop works, also for v4 destinations.
routes[].metric so our management-NIC default wins over in-band interfaces.

Aaaand … we’d need host loopback addresses on lo or loopback dummy devices, whichever fits? Then those addresses would also need to be announced by the speaker (we use strict uRPF on the leaves); is there any way yet to dot that yet, or would it be needed as part of the same potential change?

With a bit of a different angle, there might be a bigger question: with the above we’d be having a running config on IncusOS, but host egress failover is then link-down only. We also need ECMP + BFD, and I could imagine two ways to get that:

  1. Speaker import feature. bgp.peers.NAME.import=default: install a default from a named peer, and then it will be gone when the session drops. Currently gobgp already has the route, it’s a RIB send to netlink that is missing afaict. This would require no second daemon, but also the result would be no BFD.

  2. Work to hav FRR as an IncusOS application. Sysext plus a services/frr config surface, and we could look at the other ones like tailscale/linstor for guidance? It would need asn, loopback, unnumbered peers with bfd, allow-listed import, export the loopback, and what Incus hands it over iBGP at ::1. Downside to this is perhaps it will be a network-facing daemon on the host; not a problem per se, but …? We’d bind to peer interfaces and bgpd+bfdd only.

We lean towards trying to work with the FRR route because it’s what actually delivers our network fabric design, and we’d rather carry it upstream than in our own image builds if more people are interested. What do you think about this?

2

For IncusOS, adding metric should be fine, feel free to send a PR. I don’t think that dev is really required though as the routes are already part of an interface definition. There is a PR open right now to add on-link support which may be giving you what you want as far as routing through a link-local IPv6 on the interface.

3

For the rest, I agree that having a good way to do L3 BGP routing all the way to the compute host would be good and in those scenarios, go-bgp can come up short a bit. You mentioned the lack of BFD which is part of the problem, the rest would be mostly Incus limitations where it doesn’t really make sense for us to expose every single BGP option under the sun to our network configuration. Advertising non-Incus routes feels a bit out of place here.

I think the FRR approach does make sense, likely shipping it as its own application rather than as a service, so closer to what we’ve done with openfga than the tailscale/netbird case. Applications get their own dedicated storage, backup/restore mechanism, config and set of actions.

We have existing logic not to immediately look for updates and such if specific applications are installed (operations-center), so the same would apply to a system with the frr application installed as we’d want to give some time for that application to startup before going to look for updates.

I suspect an ideal setup would then be un-numbered sessions to the two ToR switches, receive the IPv4 and IPv6 default gateways that way, advertise a local IP (can be in the FRR config directly), peer with Incus itself through localhost to pick up the Incus advertisements. And setup BFD sessions with the ToR switches.

We also still need to figure out the details of Support BGP-EVPN with VXLAN · Issue #2738 · lxc/incus · GitHub as there’s definitely a world in which we give up on our built-in go-bgp for that one and instead decide to rely on a local FRR for that instead. Effectively limiting the use of the built-in go-bgp server for cases where we don’t need interactions with the kernel/netlink.

4

Thanks, that clears up my understanding and sets direction, awesome! Agree on dev, the on-link PR should cover our case for that and I had not seen it yet. We’ll send the metric one.

We’ll start (slowly …) to do some work for frr as an application, with the responsibility split as you describe: Incus gobgp announces what it knows, frr owns the host routing (defaults, loopback, bfd), and peering would then happen over localhost. This is 100% inline with our k8s bare metal setup so its great!

Should we write a short spec for the app config and post it here to discuss before starting? We would try to make sure it fits how openfga/operations-center are done thanks for those hints. Question: does the app model let a sysext ship its own networkd units, to have the app create the loopback dummy itself?