LXD, OVN and BGP - routes missing

I’m new to BGP and I might be missing just something tiny - but I seem unable to figure it out. I followed https://www.youtube.com/watch?v=1M__Rm9iZb8&t and https://linuxcontainers.org/lxd/docs/master/howto/network_bgp/.

It seems that the routes are not announced to the router (pfsense). I see the bgp session, but no new routes.

lxc network list

+-----------------+----------+---------+----------------+---------------------------+-------------+---------+---------+
|      NAME       |   TYPE   | MANAGED |      IPV4      |           IPV6            | DESCRIPTION | USED BY |  STATE  |
+-----------------+----------+---------+----------------+---------------------------+-------------+---------+---------+
| UPLINKG         | physical | YES     |                |                           |             | 1       | CREATED |
+-----------------+----------+---------+----------------+---------------------------+-------------+---------+---------+
| br-int          | bridge   | NO      |                |                           |             | 0       |         |
+-----------------+----------+---------+----------------+---------------------------+-------------+---------+---------+
| enp35s0         | physical | NO      |                |                           |             | 0       |         |
+-----------------+----------+---------+----------------+---------------------------+-------------+---------+---------+
| enp36s0         | physical | NO      |                |                           |             | 0       |         |
+-----------------+----------+---------+----------------+---------------------------+-------------+---------+---------+
| enp43s0f0       | physical | NO      |                |                           |             | 0       |         |
+-----------------+----------+---------+----------------+---------------------------+-------------+---------+---------+
| enp43s0f1       | physical | NO      |                |                           |             | 1       |         |
+-----------------+----------+---------+----------------+---------------------------+-------------+---------+---------+
| enx1617aaf827cb | physical | NO      |                |                           |             | 0       |         |
+-----------------+----------+---------+----------------+---------------------------+-------------+---------+---------+
| lxdfan0         | bridge   | YES     |                |                           |             | 1       | CREATED |
+-----------------+----------+---------+----------------+---------------------------+-------------+---------+---------+
| lxdovn2         | bridge   | NO      |                |                           |             | 0       |         |
+-----------------+----------+---------+----------------+---------------------------+-------------+---------+---------+
| my-ovn          | ovn      | YES     | 10.202.37.1/24 | fd42:4948:dd87:8d77::1/64 |             | 3       | CREATED |
+-----------------+----------+---------+----------------+---------------------------+-------------+---------+---------+

lxc network show UPLINKG

config:
  bgp.peers.r01.address: 172.17.13.1
  bgp.peers.r01.asn: "64514"
  dns.nameservers: 172.17.40.1
  ipv4.gateway: 172.17.40.1/22
  ipv4.ovn.ranges: 172.17.41.100-172.17.41.254
  volatile.last_state.created: "false"
description: ""
name: UPLINKG
type: physical
used_by:
- /1.0/networks/my-ovn
managed: true
status: Created
locations:
- n01
- 172.17.13.12
- 172.17.13.13

lxc network show my-ovn

config:
  bridge.mtu: "1442"
  ipv4.address: 10.202.37.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:4948:dd87:8d77::1/64
  ipv6.nat: "true"
  network: UPLINKG
  volatile.network.ipv4.address: 172.17.41.100
description: ""
name: my-ovn
type: ovn
used_by:
- /1.0/instances/u1
- /1.0/instances/u2
- /1.0/instances/u3
managed: true
status: Created
locations:
- n01
- 172.17.13.12
- 172.17.13.13

lxc list

+------+---------+--------------------+-----------------------------------------------+-----------+-----------+--------------+
| NAME |  STATE  |        IPV4        |                     IPV6                      |   TYPE    | SNAPSHOTS |   LOCATION   |
+------+---------+--------------------+-----------------------------------------------+-----------+-----------+--------------+
| u1   | RUNNING | 10.202.37.2 (eth0) | fd42:4948:dd87:8d77:216:3eff:fe27:845c (eth0) | CONTAINER | 0         | n01          |
+------+---------+--------------------+-----------------------------------------------+-----------+-----------+--------------+
| u2   | RUNNING | 10.202.37.3 (eth0) | fd42:4948:dd87:8d77:216:3eff:fe0e:37b5 (eth0) | CONTAINER | 0         | 172.17.13.12 |
+------+---------+--------------------+-----------------------------------------------+-----------+-----------+--------------+
| u3   | RUNNING | 10.202.37.4 (eth0) | fd42:4948:dd87:8d77:216:3eff:fefb:fb9d (eth0) | CONTAINER | 0         | 172.17.13.13 |
+------+---------+--------------------+-----------------------------------------------+-----------+-----------+--------------+

pfsense

IPv4 Unicast Summary:
BGP router identifier 192.168.245.2, local AS number 64514 vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 3, using 43 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
172.17.13.11    4      65100        78        78        0    0    0 00:04:01            0        0
172.17.13.12    4      65100        76        76        0    0    0 00:36:25            0        0
172.17.13.13    4      65100       288       288        0    0    0 00:04:01            0        0

Total number of neighbors 3
same neighbor info exist for .12 and .13
BGP neighbor is 172.17.13.11, remote AS 65100, local AS 64514, external link
Hostname: n01
  BGP version 4, remote router ID 172.17.13.11, local router ID 192.168.245.2
  BGP state = Established, up for 00:04:01
  Last read 00:00:01, Last write 00:00:00
  Hold time is 90, keepalive interval is 30 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    AddPath:
      IPv4 Unicast: RX advertised IPv4 Unicast
    Extended nexthop: received
      Address families by peer:
                   IPv4 Unicast
    Route refresh: advertised and received(new)
    Address Family IPv4 Unicast: advertised and received
    Address Family IPv6 Unicast: received
    Hostname Capability: advertised (name: pf01.wb.local,domain name: n/a) received (name: n01,domain name: n/a)
    Graceful Restart Capability: advertised and received
      Remote Restart timer is 120 seconds
      Address families by peer:
        IPv4 Unicast(not preserved)
  Graceful restart information:
    End-of-RIB send: IPv4 Unicast
    End-of-RIB received: IPv4 Unicast
    The remaining time of stalepath timer is 105
    Local GR Mode: Helper*
    Remote GR Mode: Restart
    R bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 120
    IPv4 Unicast:
      F bit: False
      End-of-RIB sent: Yes
      End-of-RIB sent after update: Yes
      End-of-RIB received: Yes
      Timers:
        Configured Stale Path Time(sec): 360
      Stale Path Remaining(sec): 105
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  2          2
    Notifications:          0          0
    Updates:                2          2
    Keepalives:            74         74
    Route Refresh:          0          0
    Capability:             0          0
    Total:                 78         78
  Minimum time between advertisement runs is 0 seconds

 For address family: IPv4 Unicast
  Update group 3, subgroup 3
  Packet Queue length 0
  Community attribute sent to this neighbor(large)
  Inbound path policy configured
  Outbound path policy configured
  Route map for incoming advertisements is *ALLOW-ALL
  Route map for outgoing advertisements is *ALLOW-ALL
  0 accepted prefixes

  Connections established 2; dropped 1
  Last reset 00:04:15,  No AFI/SAFI activated for peer
Local host: 172.17.13.1, Local port: 179
Foreign host: 172.17.13.11, Foreign port: 60905
Nexthop: 172.17.13.1
Nexthop global: fe80::ae1f:6bff:fe17:a02f
Nexthop local: fe80::ae1f:6bff:fe17:a02f
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 120
Estimated round trip time: 2 ms
Read thread: on  Write thread: on  FD used: 25

I captured the packets going out from LXD to r01. I see the notification, open and update messages from LXD. But It seems to me that the LXD update message contains no network information. I’m do not know the BGP protocol, but should the update message always include all routes or is it more like a diff to a previous state and it’s fine that there are no route information in the update message? - I restarted LXD previous while running the capture to get all messages.

Frame 39: 87 bytes on wire (696 bits), 87 bytes captured (696 bits)
Internet Protocol Version 4, Src: 172.17.13.12, Dst: 172.17.13.1
Transmission Control Protocol, Src Port: 46139, Dst Port: 179, Seq: 39, Ack: 39, Len: 21
Border Gateway Protocol - NOTIFICATION Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 21
    Type: NOTIFICATION Message (3)
    Major error Code: Cease (6)
    Minor error Code (Cease): Peer De-configured (3)
Frame 73: 144 bytes on wire (1152 bits), 144 bytes captured (1152 bits)
Internet Protocol Version 4, Src: 172.17.13.12, Dst: 172.17.13.1
Transmission Control Protocol, Src Port: 59889, Dst Port: 179, Seq: 1, Ack: 1, Len: 78
Border Gateway Protocol - OPEN Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 78
    Type: OPEN Message (1)
    Version: 4
    My AS: 65100
    Hold Time: 90
    BGP Identifier: 172.17.13.12
    Optional Parameters Length: 49
    Optional Parameters
        Optional Parameter: Capability
            Parameter Type: Capability (2)
            Parameter Length: 47
            Capability: Route refresh capability
                Type: Route refresh capability (2)
                Length: 0
            Capability: FQDN Capability
                Type: FQDN Capability (73)
                Length: 5
                Hostname Length: 3
                Hostname: n02
                Domain Name Length: 0
                Domain Name: 
            Capability: Multiprotocol extensions capability
                Type: Multiprotocol extensions capability (1)
                Length: 4
                AFI: IPv4 (1)
                Reserved: 00
                SAFI: Unicast (1)
            Capability: Multiprotocol extensions capability
                Type: Multiprotocol extensions capability (1)
                Length: 4
                AFI: IPv6 (2)
                Reserved: 00
                SAFI: Unicast (1)
            Capability: Support for 4-octet AS number capability
                Type: Support for 4-octet AS number capability (65)
                Length: 4
                AS Number: 65100
            Capability: Graceful Restart capability
                Type: Graceful Restart capability (64)
                Length: 10
                Restart Timers: 0x0078
                    0... .... .... .... = Restart state: No
                    .0.. .... .... .... = Graceful notification: No
                    .... 0000 0111 1000 = Time: 120
                AFI: IPv4 (1)
                SAFI: Unicast (1)
                Flag: 0x00
                    0... .... = Preserve forwarding state: No
                AFI: IPv6 (2)
                SAFI: Unicast (1)
                Flag: 0x00
                    0... .... = Preserve forwarding state: No
            Capability: Extended Next Hop Encoding
                Type: Extended Next Hop Encoding (5)
                Length: 6
                AFI: IPv4 (1)
                SAFI: Unicast (1)
                Next hop AFI: IPv6 (2)
Frame 80: 89 bytes on wire (712 bits), 89 bytes captured (712 bits)
Internet Protocol Version 4, Src: 172.17.13.12, Dst: 172.17.13.1
Transmission Control Protocol, Src Port: 59889, Dst Port: 179, Seq: 98, Ack: 111, Len: 23
Border Gateway Protocol - UPDATE Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 23
    Type: UPDATE Message (2)
    Withdrawn Routes Length: 0
    Total Path Attribute Length: 0

I found in another post the following command
lxc query /internal/testing/bgp

{
	"peers": [
		{
			"address": "172.17.13.1",
			"asn": 64514,
			"count": 1,
			"holdtime": 0,
			"password": ""
		}
	],
	"prefixes": [],
	"server": {
		"address": ":179",
		"asn": 65100,
		"router_id": "172.17.13.11",
		"running": true
	}
}

I’m guessing this is the reason - prefixes is empty - because of that the update message is empty. I have no idea why it’s empty - any hint to continue debugging would be mostly welcomed.

I figured it finally out. The missing piece was BGP Session with LXD backed by OVN - #13 by stgraber

I had to set the ipv4.routes=10.202.37.0/24 on the UPLINKG network and set ipv4.nat=false on the my-ovn network - now I see the peers via lxc query and on my router/r01.

:wink:

1 Like

Ah glad you got it sorted!