Question on using bgp with incus

Incus
1

I tried to configure bgp in my incus cluster after watching this:

There are 3 member in my incus cluster and the following is my execution process

frr router : 172.31.30.137
hci01: 172.31.30.134
hci02: 172.31.30.135
hci03: 172.31.30.136

root@hci01:~# incus --version
6.0.0

root@hci01:~# incus network create --target hci01 bgp-br
root@hci01:~# incus network create --target hci02 bgp-br
root@hci01:~# incus network create --target hci03 bgp-br
root@hci01:~# incus network create bgp-br

on hci01

root@hci01:~# incus config set core.bgp_address=172.31.30.134:179
root@hci01:~# incus config set core.bgp_asn=65100
root@hci01:~# incus config set core.bgp_routerid=172.31.30.134

on hci02

root@hci02:~# incus config set core.bgp_address=172.31.30.135:179
root@hci02:~# incus config set core.bgp_asn=65100
root@hci02:~# incus config set core.bgp_routerid=172.31.30.135

on hci03

root@hci03:~# incus config set core.bgp_address=172.31.30.136:179
root@hci03:~# incus config set core.bgp_asn=65100
root@hci03:~# incus config set core.bgp_routerid=172.31.30.136

on hci01

incus network set bgp-br bgp.peers.myfrr.address=172.31.30.137
incus network set bgp-br bgp.peers.myfrr.asn=399765

on frr

configure terminal
router bgp 399765
neighbor 172.31.30.134 remote-as 65100
neighbor 172.31.30.135 remote-as 65100
neighbor 172.31.30.136 remote-as 65100

address-family ipv4 unicast
neighbor 172.31.30.134 soft-reconfiguration inbound
neighbor 172.31.30.134 prefix-list bgp-everything in
neighbor 172.31.30.134 prefix-list bgp-nothing out
neighbor 172.31.30.134 activate
neighbor 172.31.30.135 soft-reconfiguration inbound
neighbor 172.31.30.135 prefix-list bgp-everything in
neighbor 172.31.30.135 prefix-list bgp-nothing out
neighbor 172.31.30.135 activate
neighbor 172.31.30.136 soft-reconfiguration inbound
neighbor 172.31.30.136 prefix-list bgp-everything in
neighbor 172.31.30.136 prefix-list bgp-nothing out
neighbor 172.31.30.136 activate
exit

address-family ipv6 unicast
neighbor 172.31.30.134 soft-reconfiguration inbound
neighbor 172.31.30.134 prefix-list bgp-everything in
neighbor 172.31.30.134 prefix-list bgp-nothing out
neighbor 172.31.30.134 activate
neighbor 172.31.30.135 soft-reconfiguration inbound
neighbor 172.31.30.135 prefix-list bgp-everything in
neighbor 172.31.30.135 prefix-list bgp-nothing out
neighbor 172.31.30.135 activate
neighbor 172.31.30.136 soft-reconfiguration inbound
neighbor 172.31.30.136 prefix-list bgp-everything in
neighbor 172.31.30.136 prefix-list bgp-nothing out
neighbor 172.31.30.136 activate
exit

on hci01

incus init cont-x86_64-ubuntu-22.04 container1 --network=bgp-br
incus network set bgp-br ipv4.address="172.31.30.1/24" ipv4.nat=false
incus config device override container1 eth0 ipv4.routes.external=172.31.30.100/32
incus exec container1 -- ip -4 addr add 172.31.30.100/32 dev eth0

on frr

mgmt01# show ip bgp neighbors 172.31.30.134 received-routes
BGP table version is 0, local router ID is 172.31.30.137, vrf id 0
Default local pref 100, local AS 399765
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *> 172.31.30.0/24   172.31.30.134                          0 65100 i
 *> 172.31.30.31/32  172.31.30.134                          0 65100 i

Total number of prefixes 2 (2 filtered)
mgmt01# 

mgmt01# show ip bgp neighbors 172.31.30.135 received-routes
BGP table version is 0, local router ID is 172.31.30.137, vrf id 0
Default local pref 100, local AS 399765
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *> 172.31.30.0/24   172.31.30.135                          0 65100 i

Total number of prefixes 1 (1 filtered)

mgmt01# show ip bgp neighbors 172.31.30.136 received-routes
BGP table version is 0, local router ID is 172.31.30.137, vrf id 0
Default local pref 100, local AS 399765
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *> 172.31.30.0/24   172.31.30.136                          0 65100 i

Total number of prefixes 1 (1 filtered)

on my pc which ip is 172.31.30.50

ping 172.31.30.100
PING 172.31.30.100 (172.31.30.100) 56(84) bytes of data.
From 172.31.30.50 icmp_seq=1 Destination Host Unreachable
From 172.31.30.50 icmp_seq=2 Destination Host Unreachable
From 172.31.30.50 icmp_seq=3 Destination Host Unreachable

on hci01

root@hci01:/run/incus/c1# incus query /internal/testing/bgp
{
        "peers": [
                {
                        "address": "172.31.30.137",
                        "asn": 399765,
                        "count": 1,
                        "holdtime": 0,
                        "password": ""
                }
        ],
        "prefixes": [
                {
                        "nexthop": "0.0.0.0",
                        "owner": "network_1",
                        "prefix": "172.31.30.0/24"
                },
                {
                        "nexthop": "0.0.0.0",
                        "owner": "instance_36_eth0",
                        "prefix": "172.31.30.100/32"
                }
        ],
        "server": {
                "address": "172.31.30.134:179",
                "asn": 65100,
                "router_id": "172.31.30.134",
                "running": true
        }
}

on frr

mgmt01# show bgp summary

IPv4 Unicast Summary:
BGP router identifier 172.31.30.137, local AS number 399765 VRF default vrf-id 0
BGP table version 0
RIB entries 2, using 256 bytes of memory
Peers 3, using 61 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
172.31.30.134   4      65100      2861      2871        0    0    0 13:15:52            0        0 N/A
172.31.30.135   4      65100      2854      2868        0    0    0 23:17:53            0        0 N/A
172.31.30.136   4      65100      2894      2897        0    0    0 23:18:36            0        0 N/A

Total number of neighbors 3

IPv6 Unicast Summary:
BGP router identifier 172.31.30.137, local AS number 399765 VRF default vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 3, using 61 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
172.31.30.134   4      65100      2861      2871        0    0    0 13:15:52            0        0 N/A
172.31.30.135   4      65100      2854      2868        0    0    0 23:17:53            0        0 N/A
172.31.30.136   4      65100      2894      2897        0    0    0 23:18:36            0        0 N/A

Total number of neighbors 3
mgmt01#

so what did i missed ? ping 172.31.30.100 from 172.31.30.50 should be reach but it didn’t.
I’m new to incus and bgp and don’t know how to debug.
Any help will be appreciated.

2

This indicates that your prefix-list on inbound is causing all your advertisements to be dropped.
So they’re received but they’re being filtered out and not actually applied to the router.

3

yeah you are right, when i ensure the following prefix-list is exists,
(2 filtered) is gone.

ip prefix-list bgp-everything seq 1 permit 172.31.30.16/28
ip prefix-list bgp-nothing seq 2 deny 0.0.0.0/0

Could you please take a look at my network topology,

  1. there are 4 bgp routers (FRR, incus server1, incus server2 ,incus server3) and one external home router TPLink.

  2. the Laptop and FRR and all three incus server and incus instances are in the same one network 172.31.30.1/24.

  3. instance1 and instance2 are in asn 65100. instance3 is just a docker container, it is in asn 399766.

WechatIMG1
WechatIMG11708×938 76.4 KB

here is my incus network that type of bridge:

root@hci02:~# incus network show bgp-br
config:
  bgp.peers.frr137.address: 172.31.30.137
  bgp.peers.frr137.asn: "399766"
  ipv4.address: 172.31.30.17/28
  ipv4.nat: "false"
  ipv6.address: fd42:d9ed:e4b3:b70d::1/64
  ipv6.nat: "true"
description: ""
name: bgp-br
type: bridge
used_by:
- /1.0/instances/c1
- /1.0/instances/c2
managed: true
status: Created
locations:
- hci01
- hci02
- hci03

and my instance 1:

root@hci02:~# incus config show c1
architecture: x86_64
config:
  image.architecture: amd64
  image.description: system_reserved
  image.name: cont-x86_64-ubuntu-22.04-origin-nogpu
  image.os: ubuntu
  image.release: jammy
  image.rootFsSize: 10GiB
  image.serial: "20240320_21:16"
  image.variant: cloud
  volatile.base_image: 516187f09902278e5b0cff006107de31485df7319090e3e2d21eb0e23f6c03a6
  volatile.cloud-init.instance-id: cee60d49-166d-490e-a13e-516b91b4ea37
  volatile.eth0.host_name: veth65e20de6
  volatile.eth0.hwaddr: 00:16:3e:03:ce:fc
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: 2c9f9ebb-05c6-424d-8675-29fd3eb4d184
  volatile.uuid.generation: 2c9f9ebb-05c6-424d-8675-29fd3eb4d184
devices:
  eth0:
    ipv4.routes.external: 172.31.30.22/32
    name: eth0
    network: bgp-br
    type: nic
ephemeral: false
profiles:
- default
stateful: false
description: ""
root@hci02:~#

and my instance 2:

root@hci02:~# incus config show c2
architecture: x86_64
config:
  image.architecture: amd64
  image.description: system_reserved
  image.name: cont-x86_64-ubuntu-22.04-origin-nogpu
  image.os: ubuntu
  image.release: jammy
  image.rootFsSize: 10GiB
  image.serial: "20240320_21:16"
  image.variant: cloud
  volatile.base_image: 516187f09902278e5b0cff006107de31485df7319090e3e2d21eb0e23f6c03a6
  volatile.cloud-init.instance-id: e98a3b2c-3644-4445-9ad5-ea9230b833c8
  volatile.eth0.host_name: vethd6eb65cf
  volatile.eth0.hwaddr: 00:16:3e:1d:5d:91
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: a8efb2db-6dbe-4e93-9606-93e0714e00c4
  volatile.uuid.generation: a8efb2db-6dbe-4e93-9606-93e0714e00c4
devices:
  eth0:
    ipv4.routes.external: 172.31.30.23/32
    name: eth0
    network: bgp-br
    type: nic
ephemeral: false
profiles:
- default
stateful: false
description: ""

Here are my questions：

  1. ping instance1 or instance2 from instance3 is reachable. ping instance3 from instance1 or instance2 is not reachable, why ? Is it because of ip prefix-list bgp-nothing seq 2 deny 0.0.0.0/0?

  2. ping instance2 from instance1 is not reachable too. but incus server1 and incus server2 were neighbors. Although these two instances belong to two different bridge networks. But can’t they access each other even after using BGP technology?

  3. neither instance1 nor instance2 could access public network ( ping 8.8.8.8 is not reachable). but ping 8.8.8.8 from every incus server is reachable. why?

  4. my laptop is not in any “as”, but i still need to access instance1(172.31.30.22/32) from my laptop(172.31.30.5/32).
    I have configured routes on the tplink router(when destination is 172.31.30.22/32 then set nexthop to the ip address of incus server1), but it didn’t work. May i ask Is it feasible to configure routes on the external home router so that i can access instance out of as (my TPLink router)?

expect for your reply. thank you very much.