Use FRR to connect to the bridge network of incus cluster members, how to configure multiple cluster members?

Incus
1

When I use FRR router to connect to the bridge network of multiple cluster members in the incus cluster through the bgp protocol, how to set FRR to connect to the virtual machine under the corresponding cluster member, because the bridge network of multiple cluster members has the same ipv4 segment.

The following is the configuration of my frr router:

Building configuration...

Current configuration:
!
frr version 8.2
frr defaults traditional
hostname OpenWrt
log syslog
!
password zebra
!
router bgp 398765
 bgp router-id 192.168.1.1
 neighbor 192.168.1.101 remote-as 65100
 neighbor 192.168.1.102 remote-as 65100
 neighbor 192.168.1.103 remote-as 65100
 !
 address-family ipv4 unicast
  neighbor 192.168.1.101 soft-reconfiguration inbound
  neighbor 192.168.1.101 prefix-list bgp-everything in
  neighbor 192.168.1.101 prefix-list bgp-everything out
  neighbor 192.168.1.102 soft-reconfiguration inbound
  neighbor 192.168.1.102 prefix-list bgp-everything in
  neighbor 192.168.1.102 prefix-list bgp-everything out
  neighbor 192.168.1.103 soft-reconfiguration inbound
  neighbor 192.168.1.103 prefix-list bgp-everything in
  neighbor 192.168.1.103 prefix-list bgp-everything out
 exit-address-family
exit
!
access-list vty seq 5 permit 127.0.0.0/8
access-list vty seq 10 deny any
!
ip prefix-list bgp-everything seq 1 permit 10.236.116.0/24
!
line vty
 access-class vty
exit
!
end

The following is the configuration of my bridging network

config:
  bgp.peers.myfrr.address: 192.168.1.1
  bgp.peers.myfrr.asn: "398765"
  ipv4.address: 10.236.116.1/24
  ipv4.nat: "false"
  ipv6.address: fd42:ce2c:fba1:4592::1/64
  ipv6.nat: "true"
description: ""
name: bgp-br
type: bridge
used_by: []
managed: true
status: Created
locations:
- hci01
- hci02
- hci03

The following is the configuration of my inus cluster:

config:
  bgp.peers.myfrr.address: 192.168.1.1
  bgp.peers.myfrr.asn: "398765"
  ipv4.address: 10.236.116.1/24
  ipv4.nat: "false"
  ipv6.address: fd42:ce2c:fba1:4592::1/64
  ipv6.nat: "true"
description: ""
name: bgp-br
type: bridge
used_by: []
managed: true
status: Created
locations:
- hci01
- hci02
- hci03
root@hci02:~# incus config show
config:
  cluster.https_address: hci02.service:8443
  core.bgp_asn: "65100"
  core.https_address: hci02.service:8443
  images.auto_update_interval: "0"
  loki.api.url: http://192.168.1.106:3100
  loki.instance: incus:clus-cZW0CNlrO3

The following is a list of my inus cluster members:

微信截图_20240708124943
微信截图_202407081249431655×460 179 KB

2

So you have independent managed bridges on each server using the exact same subnets and running some instances. You’re going to have a bad time with that.

As you noticed, this will result in each server just advertising the IPv4 /24 and IPv6 /64 which will be the same on all servers so your machine running FRR will assume that it can reach any of the servers to route traffic to an instance within that subnet.

This obviously won’t work. You could use a bit of a workaround and put a specific /32 in ipv4.routes.external for each specific instance, therefore having that instance advertise that specific route.

But if you do that, while you may get working routes for your FRR machine, you’ll have the problem that an instance on the first server won’t be able to talk to an instance on the second server, as they’ll think they’re sharing the same L2 when they’re in fact not.

So basically, if no instance to instance communication is needed, then you can probably keep this setup and use ipv4.routes.external and/or ipv6.routes.external to cause instance-specific BGP advertisements.

If you do need instance to instance to work properly, then you need an actual shared L2 within your cluster, which most commonly is achieved through OVN these days.

1 Like
3

@dong.zhaonan

VxLAN can be configured for multiple Linux bridges with each bridge on its own VNI

I’d suggest reading about BGP-FRR-VxLAN here:

https://docs.frrouting.org/en/latest/evpn.html

Vincent Bernat wrote a great Guide in 2017 which is still valid from what I know.

He already focused on using BGP-FRR-VxLAN for Virtual Machines (VM) but it should work the same for Containers.

https://vincent.bernat.ch/en/blog/2017-vxlan-bgp-evpn

His Guide shows configs for VxLAN and linux bridges like incusbr0

4

I have a question after reading this document：

It is obvious that in an incus cluster, after creating a bridge network, this bridge network can be used for any instance that running on any incus member. And by default, each server using the exact same subnets. May I ask why they are the same? Or could I change them to be different?

About the usage scenarios of BGP, if I only have one incus member, i can use bridge network.
if i have multiple incus member, I should use ovn network for bgp peers instead of bridge network.
Is this understanding correct?