BGP Session with LXD backed by OVN

LXD
,
1

Hello LXD Fam!

Need a little bit of clarity as I’m trying to set up the BGP server on lxd to talk to my EdgeRouter network device. LXD seems simple enough to set up and I believe that is working as it should. When I configure my router to do BGP. I see the router trying to stand up a session but LXD doesn’t seem to like it. Likely a better explanation is that I’m missing a configuration on the edgerouter side. I’m just trying to figure out what LXD is looking for.

Context: I have OVN configured and working. I want to move this forward to announce networks that get created to to my router so my other networks can reach them.

LXC/LXD version : 5.0.0

LXC Config `lxc config set core.bgp_address=192.168.20.50:179
lxc config set core.bgp_asn=65100
lxc config set core.bgp_routerid=192.168.20.50`

Here is the error:  `Jul 13 17:23:09 LABHOST-01 lxd.daemon[158353]: time="2022-07-13T17:23:09Z" level=info msg="Can't find configuration for a new passive connection" Key=192.168.20.1 Topic=Peer`

Thank you!

2

The commands above would enable the BGP listener, but you need to add your peers.

As you’re using OVN, you’ll need to add the BGP peers to your uplink/parent network.
Something like:

stgraber@dakara:~$ lxc network show s-dcmtl-cluster:UPLINK --project default
config:
  bgp.peers.frr01.address: 2602:fc62:a:101::10
  bgp.peers.frr01.asn: "399760"
  bgp.peers.frr02.address: 2602:fc62:a:101::11
  bgp.peers.frr02.asn: "399760"
  bgp.peers.frr03.address: 2602:fc62:a:101::12
  bgp.peers.frr03.asn: "399760"
[SNIP]
managed: true
status: Created
locations:
- abydos
- langara
- orilla
3

Thank you and I’m sorry. I should of put this in from before @stgraber! I did add the peers to the UPLINK network of Ovn

config:
  bgp.peers.edgerouter.address: 192.168.20.1
  bgp.peers.edgrouter.asn: "65200"
{redacted}
description: ""
name: UPLINK
type: physical
used_by:
- /1.0/networks/demo-01?project=demo
- /1.0/networks/my-ovn
managed: true
status: Created
locations:
- labhost-01
- labhost-03
- labhost-02

It does look like the router is trying to communicate with lxd but giving that config error. Also this is a cluster of 3 host. Do I need to add all 3 hosts as BGP peers?

Thank you

4

Yeah, you’ll need to do the lxc config set bits on all 3 hosts and will need to add all 3 to your router as they won’t be announcing quite the same thing.

5

If all the IPs and ASNs look right, you may want to do a systemctl reload snap.lxd.daemon on all 3 servers to see if some config maybe didn’t get picked up properly.

6

I set all of the config bits on all 3 host and added them as neighbors on the router, and restarted the daemon but still the same error on all 3 hosts now .

level=info msg="Can't find configuration for a new passive connection" Key=192.168.20.1 Topic=Pee

Which i’m starting to suspect maybe the router is communicating correctly with lxd but it’s either sending or not sending something and lxd is not establishing the communication.

7

Might be useful to tcpdump the traffic and see what’s going on.
Normally you should see both sides trying to establish the session.

One thing I’ve seen as an issue in the past is misaligned source addresses causing one side to ignore incoming connections. Though it doesn’t seem to be the issue here as the incoming address matches the one in your peers.

You can do lxc query /internal/testing/bgp to see the state of LXD’s BGP server.
It should look something like:

{
	"peers": [
		{
			"address": "2602:fc62:b:1000::2",
			"asn": 399760,
			"count": 2,
			"password": ""
		},
		{
			"address": "2602:fc62:b:1000::3",
			"asn": 399760,
			"count": 2,
			"password": ""
		}
	],
	"prefixes": [
		{
			"nexthop": "::",
			"owner": "network_7",
			"prefix": "2602:fc62:b:251::/64"
		},
		{
			"nexthop": "0.0.0.0",
			"owner": "network_1",
			"prefix": "172.17.250.0/24"
		},
		{
			"nexthop": "::",
			"owner": "network_1",
			"prefix": "2602:fc62:b:250::/64"
		}
	],
	"server": {
		"address": ":179",
		"asn": 399760,
		"router_id": "10.0.0.1",
		"running": true
	}
}

Though yours will have relevant nexthop as you’re running OVN whereas the example above is what I have on my desktop computer using a normal LXD bridge.

8

I don’t get anywhere near the amount of information you have in yours. I’m assuming cause it hasn’t set up a peer session yet just the server stuff.

lxc query /internal/testing/bgp
{
	"peers": [],
	"prefixes": [],
	"server": {
		"address": "192.168.20.50:179",
		"asn": 64512,
		"router_id": "192.168.20.50",
		"running": true
	}
}

tcpdump looks like it’s coming from the right source.

04:00:07.043932 IP 192.168.20.1.37390 > LABHOST-01.thelabs.online.bgp: Flags [S], seq 2978865443, win 29200, options [mss 1460,sackOK,TS val 83324845 ecr 0,nop,wscale 7], length 0
04:00:07.044012 IP LABHOST-01.thelabs.online.bgp > 192.168.20.1.37390: Flags [S.], seq 2657704582, ack 2978865444, win 65160, options [mss 1460,sackOK,TS val 805056268 ecr 83324845,nop,wscale 7], length 0
04:00:07.044540 IP 192.168.20.1.37390 > LABHOST-01.thelabs.online.bgp: Flags [.], ack 1, win 229, options [nop,nop,TS val 83324845 ecr 805056268], length 0
04:00:07.044540 IP 192.168.20.1.37390 > LABHOST-01.thelabs.online.bgp: Flags [P.], seq 1:54, ack 1, win 229, options [nop,nop,TS val 83324845 ecr 805056268], length 53: BGP
04:00:07.044644 IP LABHOST-01.thelabs.online.bgp > 192.168.20.1.37390: Flags [.], ack 54, win 509, options [nop,nop,TS val 805056269 ecr 83324845], length 0
04:00:07.044933 IP LABHOST-01.thelabs.online.bgp > 192.168.20.1.37390: Flags [R.], seq 1, ack 54, win 509, options [nop,nop,TS val 805056269 ecr 83324845], length 0

Do you think because when LXD is talking to the router it’s coming from LABHOST-01.thelabs.online.bgp?. Which I’m not sure what that is.

And for the sake of it, my routers bgp summary

show ip bgp summary
BGP router identifier 192.168.20.1, local AS number 64512
BGP table version is 1
0 BGP AS-PATH entries
0 BGP community entries

Neighbor                 V   AS   MsgRcv    MsgSen TblVer   InQ   OutQ    Up/Down   State/PfxRcd
192.168.20.50            4 64512    0        175       0      0      0     never      Active

Total number of neighbors 1

Total number of Established sessions 0

Don’t mind the same ASN numbers… this was just me trying to take shots in the dark.

9

@stgraber Just for a little more clarity! I spinned up a container on my local network using lxd and installed Bird-BGP and was able to successfully establish a tunnel with my router. So I believe my router is working exactly as it should. I’m not sure what I’m not doing on the lxd side.

Adrean@BlackBox:~$ show ip bgp summary
BGP router identifier 192.168.20.1, local AS number 64512
BGP table version is 1
0 BGP AS-PATH entries
0 BGP community entries

Neighbor                 V   AS   MsgRcv    MsgSen TblVer   InQ   OutQ    Up/Down   State/PfxRcd
192.168.20.50            4 64512    0        601       0      0      0     never      Active
192.168.20.228           4 65200    3          2       1      0      0  00:00:04               0

Total number of neighbors 2

Total number of Established sessions 1

192.168.20.50 is lxd that isn’t coming up
192.168.20.228 is the lxd container with bird-bgp with the session established. Could I have misconfigured ovn or something? would it matter that my physical network lives on a bridge on my lXD host?

Also since i’m using ovn do I have to set the lxd bgp server address to the routers on ovn lxd UPLINK network and not the lxd host itself?

11 
config:
  bgp.peers.edgerouter.address: 192.168.20.1
  bgp.peers.edgrouter.asn: "65200"

You have a typo here, the second sohuld be edgerouter not edgrouter, the fact that you don’t have both entries set on the same peer would explain why LXD isn’t setting up that session.

12

Oh my gawd!!!

{
	"peers": [
		{
			"address": "192.168.20.1",
			"asn": 65200,
			"count": 1,
			"password": ""
		}
	],
	"prefixes": [],
	"server": {
		"address": "192.168.20.50:179",
		"asn": 64512,
		"router_id": "192.168.20.50",
		"running": true
	}
}

One more question. Figured I should ask here. My assumption ( probably wrong) that once we establish a peering session the ovn networks created on the host(s) would get advertised to the router with the next hops and all that automatically. If that is not the case, how do I get these networks advertised so the rest of my physical networks can talk to them… Or does it not work like that at all ?

Thanks again for all the help. I can’t believe it was an e that got me.

13

You should set ovn.ingress_mode to routed on your uplink network.

Additionally, LXD will advertise routes for networks that do not use NAT, if you’re using ipv4.nat, then the IPv4 subnet won’t be advertised, same for IPv6 with ipv6.nat.

External addresses/subnets directly added to an instance nic through ipv4.routes.external or ipv6.routes.external will also be automatically advertised through BGP.

2 Likes
14

Oh I see! I misunderstood the use-case. This is use usually used if you are running a public service in lxd and want it to be routed to lxd.

So if I wanted to expose an internal lxd network ex. 10.186.x.x to a physical internal network ex. 192.168.x.x. Using this BGP feature wouldn’t be the right way to do it.

Sorry just trying to wrap my head around how this all works.

15

It should work fine, you’ll just have to make sure that your EdgeRouter is the one handling NAT (which it likely already is).

16

Oh awesome. Only place I’m doing any natting is at the EdgeRouter!!! I do have a physical switch between the router and my lxd host but that’s doing vlan routing only.

Trying this out. I went to create a network with nat off.

lxc network create demo-01 ipv4.nat=false --type=ovn network=UPLINK

lxd complained that my physical network doesn’t have the routes.


Uplink network doesn't contain "10.155.254.0/24" in its routes

Would i need to use something like ipv4.routesin the uplink network to make this work?

17

Yup!! It works.

I needed to add the ipv4.routes= to the uplink network and then create the network with ipv4.nat=false and whatever subnet you added to the uplink network. BGP did it’s thing and I can get to the instances on that network.

Thank you @stgraber for walking this down with me! I truly do appreciate you.

1 Like
18

Yes ipv(n).routes restricts what is allowed to be announced by lxd on the uplink.

19

@tomp that makes a ton of sense! Thank you so much! Documentation around this would be very helpful. If there is and I missed it my bad!

Thanks again both of you! learned lots!