Question on using bgp with incus

Incus
1

I tried to configure bgp in my incus cluster after watching this:

There are 3 member in my incus cluster and the following is my execution process

frr router : 172.31.30.137
hci01: 172.31.30.134
hci02: 172.31.30.135
hci03: 172.31.30.136

root@hci01:~# incus --version
6.0.0

root@hci01:~# incus network create --target hci01 bgp-br
root@hci01:~# incus network create --target hci02 bgp-br
root@hci01:~# incus network create --target hci03 bgp-br
root@hci01:~# incus network create bgp-br

on hci01

root@hci01:~# incus config set core.bgp_address=172.31.30.134:179
root@hci01:~# incus config set core.bgp_asn=65100
root@hci01:~# incus config set core.bgp_routerid=172.31.30.134

on hci02

root@hci02:~# incus config set core.bgp_address=172.31.30.135:179
root@hci02:~# incus config set core.bgp_asn=65100
root@hci02:~# incus config set core.bgp_routerid=172.31.30.135

on hci03

root@hci03:~# incus config set core.bgp_address=172.31.30.136:179
root@hci03:~# incus config set core.bgp_asn=65100
root@hci03:~# incus config set core.bgp_routerid=172.31.30.136

on hci01

incus network set bgp-br bgp.peers.myfrr.address=172.31.30.137
incus network set bgp-br bgp.peers.myfrr.asn=399765

on frr

configure terminal
router bgp 399765
neighbor 172.31.30.134 remote-as 65100
neighbor 172.31.30.135 remote-as 65100
neighbor 172.31.30.136 remote-as 65100

address-family ipv4 unicast
neighbor 172.31.30.134 soft-reconfiguration inbound
neighbor 172.31.30.134 prefix-list bgp-everything in
neighbor 172.31.30.134 prefix-list bgp-nothing out
neighbor 172.31.30.134 activate
neighbor 172.31.30.135 soft-reconfiguration inbound
neighbor 172.31.30.135 prefix-list bgp-everything in
neighbor 172.31.30.135 prefix-list bgp-nothing out
neighbor 172.31.30.135 activate
neighbor 172.31.30.136 soft-reconfiguration inbound
neighbor 172.31.30.136 prefix-list bgp-everything in
neighbor 172.31.30.136 prefix-list bgp-nothing out
neighbor 172.31.30.136 activate
exit

address-family ipv6 unicast
neighbor 172.31.30.134 soft-reconfiguration inbound
neighbor 172.31.30.134 prefix-list bgp-everything in
neighbor 172.31.30.134 prefix-list bgp-nothing out
neighbor 172.31.30.134 activate
neighbor 172.31.30.135 soft-reconfiguration inbound
neighbor 172.31.30.135 prefix-list bgp-everything in
neighbor 172.31.30.135 prefix-list bgp-nothing out
neighbor 172.31.30.135 activate
neighbor 172.31.30.136 soft-reconfiguration inbound
neighbor 172.31.30.136 prefix-list bgp-everything in
neighbor 172.31.30.136 prefix-list bgp-nothing out
neighbor 172.31.30.136 activate
exit

on hci01

incus init cont-x86_64-ubuntu-22.04 container1 --network=bgp-br
incus network set bgp-br ipv4.address="172.31.30.1/24" ipv4.nat=false
incus config device override container1 eth0 ipv4.routes.external=172.31.30.100/32
incus exec container1 -- ip -4 addr add 172.31.30.100/32 dev eth0

on frr

mgmt01# show ip bgp neighbors 172.31.30.134 received-routes
BGP table version is 0, local router ID is 172.31.30.137, vrf id 0
Default local pref 100, local AS 399765
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *> 172.31.30.0/24   172.31.30.134                          0 65100 i
 *> 172.31.30.31/32  172.31.30.134                          0 65100 i

Total number of prefixes 2 (2 filtered)
mgmt01# 

mgmt01# show ip bgp neighbors 172.31.30.135 received-routes
BGP table version is 0, local router ID is 172.31.30.137, vrf id 0
Default local pref 100, local AS 399765
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *> 172.31.30.0/24   172.31.30.135                          0 65100 i

Total number of prefixes 1 (1 filtered)

mgmt01# show ip bgp neighbors 172.31.30.136 received-routes
BGP table version is 0, local router ID is 172.31.30.137, vrf id 0
Default local pref 100, local AS 399765
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *> 172.31.30.0/24   172.31.30.136                          0 65100 i

Total number of prefixes 1 (1 filtered)

on my pc which ip is 172.31.30.50

ping 172.31.30.100
PING 172.31.30.100 (172.31.30.100) 56(84) bytes of data.
From 172.31.30.50 icmp_seq=1 Destination Host Unreachable
From 172.31.30.50 icmp_seq=2 Destination Host Unreachable
From 172.31.30.50 icmp_seq=3 Destination Host Unreachable

on hci01

root@hci01:/run/incus/c1# incus query /internal/testing/bgp
{
        "peers": [
                {
                        "address": "172.31.30.137",
                        "asn": 399765,
                        "count": 1,
                        "holdtime": 0,
                        "password": ""
                }
        ],
        "prefixes": [
                {
                        "nexthop": "0.0.0.0",
                        "owner": "network_1",
                        "prefix": "172.31.30.0/24"
                },
                {
                        "nexthop": "0.0.0.0",
                        "owner": "instance_36_eth0",
                        "prefix": "172.31.30.100/32"
                }
        ],
        "server": {
                "address": "172.31.30.134:179",
                "asn": 65100,
                "router_id": "172.31.30.134",
                "running": true
        }
}

on frr

mgmt01# show bgp summary

IPv4 Unicast Summary:
BGP router identifier 172.31.30.137, local AS number 399765 VRF default vrf-id 0
BGP table version 0
RIB entries 2, using 256 bytes of memory
Peers 3, using 61 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
172.31.30.134   4      65100      2861      2871        0    0    0 13:15:52            0        0 N/A
172.31.30.135   4      65100      2854      2868        0    0    0 23:17:53            0        0 N/A
172.31.30.136   4      65100      2894      2897        0    0    0 23:18:36            0        0 N/A

Total number of neighbors 3

IPv6 Unicast Summary:
BGP router identifier 172.31.30.137, local AS number 399765 VRF default vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 3, using 61 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
172.31.30.134   4      65100      2861      2871        0    0    0 13:15:52            0        0 N/A
172.31.30.135   4      65100      2854      2868        0    0    0 23:17:53            0        0 N/A
172.31.30.136   4      65100      2894      2897        0    0    0 23:18:36            0        0 N/A

Total number of neighbors 3
mgmt01#

so what did i missed ? ping 172.31.30.100 from 172.31.30.50 should be reach but it didn’t.
I’m new to incus and bgp and don’t know how to debug.
Any help will be appreciated.

2

This indicates that your prefix-list on inbound is causing all your advertisements to be dropped.
So they’re received but they’re being filtered out and not actually applied to the router.

3

now the 2 filtered is gone but still didn’t work.
here is my network topology:

Untitled Diagram
Untitled Diagram1215×1040 94.8 KB

I’m not quite sure if my understanding of bgp is correct.
If my requirement is let the container in the diagram have an external network IP address(172.31.30.100) so that my laptop can access the container on the simplest network.
When ping from 172.31.30.50, TPlink Router will drop the traffic because there aren’t any routes in my tplink router that used for direct traffic to incus container.

So do I have the FRR router deployed correctly?
Does FFRouting have the ability to announce its routes to upstream tplink routers?
Should I do some kind of setting for FFRouting in the tplink router?
Just like add one route entry when dest is 172.31.30.100/32 then redirect traffic to 172.31.30.137/32?
Please tell me if there is a problem with my topology, if not then what did i missed?

my FRRouting (172.31.30.137) and my container（172.31.30.100/32）are in same network .
here is the result of route -n in frr server

root@mgmt01:~#  route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.31.30.1     0.0.0.0         UG    0      0        0 eno1
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.20.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-6d5bb14dc0c9
172.22.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-c79e67ffb540
172.31.30.0     0.0.0.0         255.255.255.0   U     0      0        0 eno1

when i ping container(172.31.30.100) from frr(172.31.30.137),
the traffic always back to frr server.
So does the frr server and container shouldn’t be deployed in one same network ?

:sweat_smile:
Forgive me for knowing too little about frr.