[LXD] OVN network to network routing

Project LXD
Status Draft
Author(s) @tomp
Approver(s) @stgraber
Release 4.19
Internal ID LXXXX

Abstract

Provide the ability to specify peering relationships between OVN networks (including across projects) so that network traffic between the OVN networks stays within the OVN subsystem and doesn’t leave OVN and then re-enter.

Rationale

Currently traffic between OVN networks exits the OVN subsystem via the source network’s virtual router and goes into the uplink network where it may then re-enter the OVN subsystem via the target network’s virtual router. This is inefficient and means that the network bandwidth is limited by the uplink network’s capabilities. If the OVN setup is using faster networking for internal traffic, then it would also be possible to use the same faster networking capabilities for OVN<->OVN traffic by allowing peering relationships to be configured between OVN networks.

Specification

Design

OVN supports creating peering links between virtual routers by adding router ports to each router and setting the peer property of the router ports to the respective router port name.

For example, to create a peering link between two existing virtual routers:

  • lxd-net1-lr (LAN subnets 10.110.120.0/24, fd42:7832:3b4e:cffb::/64)
  • lxd-net2-lr (LAN subnets 10.105.164.0/24, fd42:5389:62b9:be7c::/64)

We can create two router ports, one on each router and reference the other as the peer.

In order to avoid having to setup a separate peering subnet, the existing MAC and IPs of the virtual router’s port on the internal LAN have been used, albeit with a single-host subnet (e.g. /32 for IPv4 and /128 for IPv6). This effectively adds the same address to multiple ports on each router. This will become important when actually setting up static routes that use the peering connection, as we will need to explicitly specify the router port to use, as OVN will not be able to deduce the correct port to use for the peering link automatically.

ovn-nbctl lrp-add lxd-net1-lr lxd-net1-lr-lrp-net-2 00:16:3e:d6:73:26 10.110.120.1/32 fd42:7832:3b4e:cffb::1/128 peer=lxd-net2-lr-lrp-net-1
ovn-nbctl lrp-add lxd-net2-lr lxd-net2-lr-lrp-net-1 00:16:3e:3f:e4:9f 10.105.164.1/32 fd42:5389:62b9:be7c::1/128 peer=lxd-net1-lr-lrp-net-2

LXD will then need to setup static routes on the respective virtual routers for the peered local subnets. The static routes will need to use the target router’s IP that was added on the peering ports for the nexthop address and explicitly specify the local peering router port to use for egress traffic, e.g:

ovn-nbctl lr-route-add lxd-net1-lr 10.105.164.0/24 10.105.164.1 lxd-net1-lr-lrp-net-2
ovn-nbctl lr-route-add lxd-net1-lr fd42:5389:62b9:be7c::/64 fd42:5389:62b9:be7c::1 lxd-net1-lr-lrp-net-2
ovn-nbctl lr-route-add lxd-net2-lr 10.110.120.0/24 10.110.120.1 lxd-net2-lr-lrp-net-1
ovn-nbctl lr-route-add lxd-net2-lr fd42:7832:3b4e:cffb::/64 fd42:7832:3b4e:cffb::1 lxd-net2-lr-lrp-net-1

This will then allow traffic to flow between networks without leaving the OVN subsystem.

Route tables (avoiding asymmetric routing for NIC routes)

Because LXD’s OVN implementation supports routing additional prefixes to ovn NICs by specifying ipv{n}.routes and/or ipv{n}.routes.external this could then result in Instance NICs being configured to create packets destined for the peer network but using a source address outside of the source network’s primary subnet. This will lead to asymmetric routing (where the return packet leaves the OVN subsystem) and cause unexpected behaviour when using stateful ACLs or external firewalls.

To avoid this we will need to make available the ability to specify which route prefixes should be added to the source network’s routing table pointing toward the target peer network. This will allow the administrator of the source network to specify exactly which prefixes they want to be reachable over the peer connection.

This will also ensure that if the target network later adds a NIC level route that conflicts with addressing inside the source network that these routes are not automatically exported to the source network which would cause network disruption.

It also allows for the ability to create peer connections to multiple networks that may contain some prefixes that conflict with each other but not the source network. In this way the source network can select which prefix is reachable over which peer connection, rather tha potentially importing a set of conflicting prefixes from the multiple peer networks.

Mutual peering

Because OVN subnets are not globally unique (even within a single LXD deployment) it is possible for the same subnet to be used in multiple OVN networks. As such a peering relationship between OVN networks needs to be mutually agreed by both sides. It will also be possible for peering relationships to be established between OVN networks in different LXD projects.

Example work flow:

Create two OVN networks in each in different projects.

lxc network create ovn1 --type=ovn --project project1 \
    network=myuplink \
    ipv4.address=192.168.1.1/24

lxc network create ovn2 --type=ovn --project project2 \
    network=myuplink \
    ipv4.address=192.168.2.1/24

Initiate peer connection from ovn1 towards ovn2.
The initiator will have to know correct project and network name to succeed, and if either are incorrect a generic “not found” message will be returned that will not indicate which (project or network name) was incorrect. This is to avoid users in one project being able to enumerate existing projects or existing networks in another project.

lxc network peer create ovn1 mypeer-ovn2 project2/ovn2 --project project1
lxc network peer ls ovn1 --project project1
+-------------+-------------+---------------+----------------+---------+
| NAME        | DESCRIPTION | PEER          | PREFIXES       | STATE   |
+-------------+-------------+---------------+----------------+---------+
| mypeer-ovn2 |             | project2/ovn2 |                | PENDING |
+-------------+-------------+---------------+----------------+---------+

Confirm peer connection from ovn2 towards ovn1.
The user will have to know correct project and network name to succeed.

lxc network peer create ovn2 mypeer-ovn1 project1/ovn1 --project project2
lxc network peer ls ovn2 --project project2
+-------------+-------------+---------------+----------------+---------+
| NAME        | DESCRIPTION | PEER          | PREFIXES       | STATE   |
+-------------+-------------+---------------+----------------+---------+
| mypeer-ovn1 |             | project1/ovn1 |                | CREATED |
+-------------+-------------+---------------+----------------+---------+

lxc network peer ls ovn1 --project project1
+-------------+-------------+---------------+----------------+---------+
| NAME        | DESCRIPTION | PEER          | PREFIXES       | STATE   |
+-------------+-------------+---------------+----------------+---------+
| mypeer-ovn2 |             | project2/ovn2 |                | CREATED |
+-------------+-------------+---------------+----------------+---------+

Add route prefixes to each side.

lxc network peer route add ovn1 mypeer-ovn2 192.168.2.0/24 --project project1
lxc network peer ls ovn1 --project project1
+-------------+-------------+---------------+----------------+---------+
| NAME        | DESCRIPTION | PEER          | PREFIXES       | STATE   |
+-------------+-------------+---------------+----------------+---------+
| mypeer-ovn2 |             | project2/ovn2 | 192.168.2.0/24 | CREATED |
+-------------+-------------+---------------+----------------+---------+

lxc network peer route add ovn2 mypeer-ovn1 192.168.1.0/24 --project project2
lxc network peer ls ovn2 --project project2
+-------------+-------------+---------------+----------------+---------+
| NAME        | DESCRIPTION | PEER          | PREFIXES       | STATE   |
+-------------+-------------+---------------+----------------+---------+
| mypeer-ovn1 |             | project1/ovn1 | 192.168.1.0/24 | CREATED |
+-------------+-------------+---------------+----------------+---------+

ACL considerations

It is hoped that OVN will eventually allow us to identify traffic going to/from a peer router port and reference that in ACL rules using the peer name. As such we will ensure that peer names are usable in ACL rules when prefixed with a special character (that already cannot be used in ACL names) to indicate a specific network port subject.

Peer names will follow the same naming restrictions as ACLs:

  • Be between 1 and 63 characters long
  • Be made up exclusively of letters, numbers and dashes from the ASCII table
  • Not start with a digit or a dash
  • Not end with a dash

Currently the ACL will classify traffic on the peer connection as @external as it does with traffic going to/from the uplink network.

Although OVN itself doesn’t support identifying traffic from the peer connection as a different specific port on the internal LAN (it all appears to come from the router’s port connected to the LAN), as the peer connection has a specific set of target prefixes associated with it, we could potentially create an ACL address set containing those prefixes. We would then need to ensure that traffic from those prefixes not coming from the peer connection was dropped and any traffic coming from an address outside of that address set through the peer connection was also dropped. At that point we could be confident that any packets matching a source address in the address set for the peer connection could only have come from the peer connection itself.

This has been tested to work using router policies in OVN.

E.g. This allows packets from lxd-net2-lr’s subnet 10.105.164.0/24 arriving at lxd-net1-lr’s peer router port, and drops all other traffic arriving at the port.

ovn-nbctl lr-policy-add lxd-net1-lr 100 "ip4.src == 10.105.164.0/24 && inport == \"lxd-net1-lr-lrp-net-2\"" allow
ovn-nbctl lr-policy-add lxd-net1-lr 99 "inport == \"lxd-net1-lr-lrp-net-2\"" drop

This provides the foundations for ensuring that packets arriving at a virtual peer router port match the prefixes expected for the peer, and equally allow ensuring that packets arriving from the external virtual router port (connected to the uplink network) do not come from prefixes expected to be coming from the peer connection. In this way a named ACL address set that references the peer connection name would be able to reliably enforce policies between networks.

API changes

For the network peers feature a new API extension will be added called network_peer with the following API endpoints and structures added:

Create and edit a network peer

POST /1.0/networks/<network>/peers
PUT /1.0/networks/<network>/peers/<name>

Using the following new API structures respectively:

type NetworkPeersPost struct {
	NetworkPeerPut `yaml:",inline"`

	// Name of the peer
	// Example: project1-network1
	Name string `json:"name" yaml:"name"`

	// Name of the target project
	// Example: project1
	TargetProject string `json:"target_project" yaml:"target_project"`

	// Name of the target network
	// Example: network1
	TargetNetwork string `json:"target_network" yaml:"target_network"`
}

type NetworkPeerPut struct {
	// Description of the peer
	// Example: Peering with network1 in project1
	Description string `json:"description" yaml:"description"`

	// Peer configuration map (refer to doc/network-peers.md)
	// Example: {"user.mykey": "foo"}
	Config map[string]string `json:"config" yaml:"config"`

	// Route prefixes for peer
	Prefixes []string `json:"prefixes" yaml:"prefixes"`
}

Delete a network peer

DELETE /1.0/networks/<network>/peers/<name>

List network peers

GET /1.0/networks/network/peers
GET /1.0/networks/<network>/peers/<name>

Returns a list or single record (respectively) of this new NetworkPeer structure:

type NetworkPeer struct {
	NetworkPeerPut `yaml:",inline"`

	// Name of the peer
	// Read only: true
	// Example: project1-network1
	Name string `json:"name" yaml:"name"`

	// Name of the target project
	// Read only: true
	// Example: project1
	TargetProject string `json:"target_project" yaml:"target_project"`

	// Name of the target network
	// Read only: true
	// Example: network1
	TargetNetwork string `json:"target_network" yaml:"target_network"`

	// The state of the peer
	// Read only: true
	// Example: Pending
	Status string `json:"status" yaml:"status"`
}

CLI changes

There will be a new sub-command added to the lxc network command called peer.

E.g.

For managing peer relationships:

lxc network peer ls <network>
lxc network peer create <network> <peer name> <[target project/]target_network>
lxc network peer edit <network> <peer name>
lxc network peer set <network> <peer name> <key>=<value>...
lxc network peer unset <network> <peer name> <key>
lxc network peer get <network> <peer name> <key>
lxc network peer delete <network> <peer name> 

For managing routes over a peer relationship:

lxc network peer route add <network> <peer name> <prefix>
lxc network peer route remove <network> <peer name> <prefix>

Database changes

There will be two new tables added called networks_peers and networks_peer_config.

CREATE TABLE "networks_peers" (
	id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
	network_id INTEGER NOT NULL,
	node_id INTEGER,
	name TEXT NOT NULL,
	description TEXT NOT NULL,
	target_network_id INTEGER NOT NULL, # Derived from target project/network
	prefixes TEXT NOT NULL, # JSON field containing route prefixes
	state INTEGER NOT NULL DEFAULT 0, # 0 = pending, 1 = active
	UNIQUE (network_id, node_id, name),
	FOREIGN KEY (network_id) REFERENCES "networks" (id) ON DELETE CASCADE,
	FOREIGN KEY (node_id) REFERENCES nodes(id) ON DELETE CASCADE
);

CREATE TABLE "networks_peer_config" (
	id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
	network_peer_id INTEGER NOT NULL,
	key VARCHAR(255) NOT NULL,
	value TEXT,
	UNIQUE (network_peer_id, key),
	FOREIGN KEY (network_peer_id) REFERENCES "networks_peers" (id) ON DELETE CASCADE
);

Upgrade handling

As these are new features, no upgrade handling is required.

Further information

Research is required to understand what is possible from an ACL perspective when classifying traffic coming from a peer connection. Currently it would be classified as external traffic.

We also need to consider the possibility of asymmetric routing when using NIC ipv{n}.routes.external feature that routes external IPs directly to a NIC (without NAT), meaning that traffic can be destined for a peer connection but come from a source address that doesn’t directly belong to the source network. This would mean the return traffic would then not go via the peer connection and instead leave the OVN subsystem only to return back into it via the virtual router’s uplink port.

Looking at other cloud providers today it seems they all have the concept of a network routing table, and require that after setting up the peer relationship that the routing tables on both sides of the relationship are updated to specify exactly which prefixes are routed between them.

This makes me wonder if perhaps we need to add a concept of network routes (e.g. lxc network route <network>) and have the admin of the OVN network configure them after setting up the peer relationship.

This would allow them to specify any far side NIC route prefixes (with the option of only using some/none of them as needed). It would also mean if peering with multiple networks, and one of them happened to have a NIC route prefix that conflicted with another peered network’s primary subnet, the admin could then select exactly which routes they want to go over which peering link.

This feels safer and more flexible than automatically exporting all NIC level prefixes to all peered networks, as that would allow far side to arbitrarily add routes to the peered networks, which seems like asking for trouble.

@stgraber further to my research regarding mutual peering with other cloud providers today, do you think it would be safe enough to the lxc network peer create <network> <peer name> <[target project/]target_network> command an error if the combination of project and network name was not found (but not state to the user which aspect was not found).

This way you’d have to correctly guess both the project name and network name in order to enumerate the projects or networks.

Otherwise we probably need something like a peering UUID that can be generated, exchanged and then used to setup/validate the peering relationship.

@stgraber ready for review - main question points are:

  1. Is having to specify project name and network name for peer connection on both sides sufficient protection from cross-project enumation?
  2. Is the propose route prefixes approach OK (it is similar to what the other cloud providers do) and enables more fine grained control over which prefixes are acceptable from the peer. Which prevents the peer from injecting other prefixes in the future.

Feels to me like restricting the exact subnets being imported/exported is something we could do through config keys or as an extra attribute of a peering later on with the default being to expose both sides as they are.

We could in theory also extend this to allow for subnet mapping (NAT) as it’s also something cloud providers often support, as much as I may hate it…

For the lxc network peer add case, I thought we discussed effectively making:

lxc network peer add lxdbr0 other-net blah/other-net

To always succeed but result in an entry in lxc network peer list lxdbr0 with a state of PENDING until such time as the other end has similarly added the peer.

This shouldn’t allow for any information leakage (neither side can tell what exists on the other) and should be pretty straightforward to implement.

Was there a problem with this approach?

I think it’d be better to treat that as out of scope for now, with the initial implementation routing everything available on the target.

We have enough flexibility with the API and config here to be able to add such restrictions on top of it, effectively then restricting the peering to a specific set of subnets with each side having control on what they’d want to export and import.

1 Like

The only thing I was concerned about was that a typo in either project or net name would potentially be difficult to identify why the peering wasn’t working. But apart from that its fine, we can just store the requested details in a field as we discussed previously.

Should mention that internal and external will to be allowed to avoid conflicting with built-in names.

Yeah so the ‘@’ prefix is currently used for the two special reserved names internal and external. The port groups belonging to members of an acl are just referenced directly without an ‘@’ so we can use the ‘@’ to indicate a particular ‘peer’ connection, with those two words being reserved.

This will then mean ACLs themselves can potentially use the same names as a peer connection.

Ok. I’m not very keen on allowing information leakage by allowing this one thing to go look at a project which they don’t have access to. Using some kind of UUID or the like as a token also would still have the same problem. To prevent potential brute-force, we’d need them completely random and would need to not disclose validity.

So it feels like this is a case for good documentation, basically having the network peering doc prominently mention that if after adding on both sides, the peering is still listed as PENDING to go take a very close look at the name of both project and network for any typo that may have been made.

OK make sense.

Right, that’s what I was thinking. We’d use @NAME for traffic source/destination which isn’t coming from a network ACL on the current network.

Assuming enough OVN plumbing, I can see us adding @my-peer/its-acl though, so we can identity traffic coming or heading towards a specific set of instances based on ACL. But we’re currently a long way from having what we need in OVN for that :slight_smile: