Abstract
The aim of this project is to add the ability for instances to have external IP addresses forwarded to them, both for bridged and ovn network types.
We also want to allow ovn networks to be able to specify a different outbound NAT address than their external assigned IP on the uplink network.
The three desired features are:
- One-to-one forward of entire single external IP to single internal IP.
- One-to-many sharing of a single external IP forwarding to multiple internal IPs via specific port(s) forwards to a single internal IP and port(s).
- Allow
ovnnetworks to specify their outbound source NAT IP from an external address on the uplink network (outside of the uplink network’s subnet).
Terminology
The following terms will be used both in this document and in the resulting API and CLI commands.
- External IP address - an IP address that is outside of the network’s own subnet and routed to the parent network’s uplink. These are the addresses that we want to be able to forward to an instance’s internal IP(s).
- Internal IP address - an existing IP address on an instance NIC that is within the network’s subnet. Can be either the primary allocated static address or a dynamic address added within the instance.
- Uplink - an existing LXD network that is connected to the network’s router to provide external connectivity. These networks also define what external IP subnets are available for use to forward IPs into instances via their
ipv{n}.routessetting. - Project
restricted.networks.subnets- A project can also define a subset of the uplink’s external subnets that are allowed for use as external IPs in the networks within that project.
Rationale
We already support forwarding external IPs from the uplink network directly into an instance NIC (without being rewritten to a internal IP destination) using the bridged NIC’s ipv{n}.routes setting or the ovn NIC’s ipv{n}.routes.external (which will also be coming to bridged NICs soon by way of [LXD] BGP address/route advertisement).
However this requires fully allocating an entire IP address to a single instance, and it requires that the instance has the external IP added as an alias inside it (which requires additional setup inside the instance by the user).
Allowing a user to forward an external IP (or selected ports of it) to an existing internal IP will allow more efficient use of the limited public IPv4 addresses available as well as simplifying the setup of providing external inbound connectivity so that there is no need to modify the network settings inside the container (i.e having to set up an IP alias for the external IP as described above).
The rationale to providing the ability to specify the outbound source NAT address for ovn networks is to provide feature parity with bridge networks, by allowing an ovn network to specify a source NAT address in one of the uplink network’s ipv{n}.routes external subnets. As currently the ovn virtual routers can only use IPs for source NAT that are in the uplink network’s own subnet.
Specification
Design
The decision to make external IP forwards be associated to a network rather than a particular instance NIC device was intentional. The idea being that because both bridge and ovn networks allow internal IPs to be used in an adhoc dynamic fashion, it maybe that an internal IP is moved between instances without the knowledge of LXD. By allowing an IP forward to be created at the network level and forwarded to a specific internal IP, the external IP forward will “follow” the internal IP even if it is moved.
Also for port-based forwards (unlike the existing NIC level ipv{n}.routes setting), there can be multiple instance NICs effectively using the same external IP, so defining it agains an instance NIC didn’t make a lot of sense.
So keeping all the external->internal forwarding features together at the network level seemed sensible.
For the ovn network type we intend to use the Load_Balancer functionality provided by OVN.
This provides for both whole IP and individual port forwarding (of either TCP or UDP). It also allows for loopback requests from the target instance to the external IP/port and handles source NAT (mostly) correctly (see below).
For the bridge network type we can extend the existing firewall package (along with the associated xtables and nftables drivers) to provide forwarding functionality using DNAT rules, and loopback support using br_netfilter, hairpin bridge ports and SNAT rules (similar to what we do today with proxy devices in NAT mode).
API changes
For the network forwards feature a new API extension will be added called network_forward with the following API endpoints and structures added:
Create and edit a network forward
POST /1.0/networks/<network>/forwards
PUT /1.0/networks/<network>/forwards/<listen_address>
Using the following new API structures respectively:
type NetworkForwardsPost struct {
NetworkForwardPut `yaml:",inline"`
// The listen address of the forward
// Example: 192.0.2.1
ListenAddress string `json:"listen_address" yaml:"listen_address"`
}
type NetworkForwardPut struct {
// Description of the forward listen IP
// Example: My public IP forward
Description string `json:"description" yaml:"description"`
// Forward configuration map (refer to doc/network-forwards.md)
// Example: {"user.mykey": "foo"}
Config map[string]string `json:"config" yaml:"config"`
// Port forwards (optional)
Ports []NetworkForwardPort `json:"ports" yaml:"ports"`
}
type NetworkForwardPort struct {
// Description of the forward port
// Example: My web server forward
Description string `json:"description" yaml:"description"`
// Protocol for port forward (either tcp or udp)
// Example: tcp
Protocol string `json:"protocol" yaml:"protocol"`
// ListenPort(s) to forward (comma delimited ranges)
// Example: 80,81,8080-8090
ListenPort string `json:"listen_port" yaml:"listen_port"`
// TargetPort(s) to forward ListenPorts to (allows for many-to-one)
// Example: 80,81,8080-8090
TargetPort string `json:"target_port" yaml:"target_port"`
// TargetAddress to forward ListenPorts to
// Example: 198.51.100.2
TargetAddress string `json:"target_address" yaml:"target_address"`
}
Delete a network forward
DELETE /1.0/networks/<network>/forwards/<listen_address>
List network forwards
GET /1.0/networks/network/forwards
GET /1.0/networks/<network>/forwards/<listen_address>
Returns a list or single record (respectively) of this new NetworkForward structure:
type NetworkForward struct {
NetworkForwardPut `yaml:",inline"`
// The listen address of the forward
// Example: 192.0.2.1
ListenAddress string `json:"listen_address" yaml:"listen_address"`
// What cluster member this record was found on
// Example: lxd01
Location string `json:"location" yaml:"location"`
}
OVN source NAT
For the OVN network source NAT address feature a new API extension called network_ovn_nat_address will be added to align with the earlier network_nat_address which added the feature to the bridge network type.
Because we intend for the ipv{n}.nat.address setting to be set to an address in the uplink network’s ipv{n}.routes external subnets, it will be outside of the uplink network’s own subnet and thus it will be needed that the specified source NAT address (or a wider subnet) is routed to the OVN network’s virtual router. As such this we will require that the uplink network’s ovn.ingress_mode be set to routed and the required routes on the uplink network have been added (either manually or via BGP).
CLI changes
For external IP forwarding there will be a new sub-command added to the lxc network command called forward.
E.g.
lxc network forward create <network> <listen_address> [key=value...] [--target=<member>]
lxc network forward port add <network> <listen_address> <protocol> <listen_port(s)> <target_address> [<target_port(s)>] [--target=<member>]
lxc network forward port remove <network> <listen_address> [<protocol>] [<listen_port(s)>] [--force] [--target=<member>]
lxc network forward delete <network> <listen_address> [--target=<member>]
lxc network forward show <network> <listen_address> [--target=<member>]
lxc network forward edit <network> <listen_address> [--target=<member>]
lxc network forward set <network> <key>=<value>... [--target=<member>]
lxc network forward unset <network> <key> [--target=<member>]
lxc network forward get <network> <key> [--target=<member>]
lxc network forward list <network>
The forward config keys available will be user.* and target_address which will set the default target address for traffic not matched by any of the port overrides. This field is optional and if not set then unmatched traffic will be dropped.
For lxc network forward port remove the --force flag will remove all port forwards that match, without if multiple port forwards match an error will be returned.
For the OVN network source NAT address feature we will add two new config keys:
ipv4.nat.addressipv6.nat.address
E.g. lxc network set <network> ipv4.nat.address=192.0.2.1
Database changes
There will be two new tables added called networks_forwards and networks_forward_config.
CREATE TABLE "networks_forwards" (
id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
network_id INTEGER NOT NULL,
node_id INTEGER,
listen_address TEXT NOT NULL,
description TEXT NOT NULL,
ports TEXT NOT NULL,
UNIQUE (network_id, node_id, listen_address),
FOREIGN KEY (network_id) REFERENCES "networks" (id) ON DELETE CASCADE,
FOREIGN KEY (node_id) REFERENCES nodes(id) ON DELETE CASCADE
);
CREATE TABLE "networks_forwards_config" (
id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
network_forward_id INTEGER NOT NULL,
key VARCHAR(255) NOT NULL,
value TEXT,
UNIQUE (network_forward_id, key),
FOREIGN KEY (network_forward_id) REFERENCES "networks_forwards" (id) ON DELETE CASCADE
);
Because each OVN network has its own virtual router connected to the uplink network, any external IPs being forwarded will need the virtual router to respond to ARP/NDP requests on its uplink interface. As such a specific listen_address can only be used on a single network at any one time (although there can be multiple per-port forwards setup on the same external IP forwarding to different instances inside the same network).
Also the IPs will be stored in canonical form so that database queries can be done on them irrespective of the format specified by the user.
Upgrade handling
As these are new features, no upgrade handling is required.
Further information
Whilst the OVN Load_Balancer meets nearly all our requirements, there does appear to be a bug in the version provided in Ubuntu Focal with regards to loopback requests from the target instance to the external IP/port.
When having a setup like below, where we have one entire external IP (fd42:b545:2e58:ec06::12) forwarding to an internal IP (fd42:3242:1613:9c39:216:3eff:fe80:6179), and one or more port based forwardings to the same target IP (fd42:3242:1613:9c39:216:3eff:fe80:6179) but listening on a different external IP (fd42:b545:2e58:ec06::11):
E.g.
ovn-nbctl --may-exist lb-add test '[fd42:b545:2e58:ec06::12]' '[fd42:3242:1613:9c39:216:3eff:fe80:6179]'
ovn-nbctl --may-exist lb-add test '[fd42:b545:2e58:ec06::11]:80' '[fd42:3242:1613:9c39:216:3eff:fe80:6179]:80' tcp
ovn-nbctl --may-exist lb-add test '[fd42:b545:2e58:ec06::11]:81' '[fd42:3242:1613:9c39:216:3eff:fe80:6179]:80' tcp
When making a loopback request from the target IP (fd42:3242:1613:9c39:216:3eff:fe80:6179), the source address of the looped back request to [fd42:b545:2e58:ec06::11]:80 is incorrectly being rewritten to fd42:b545:2e58:ec06::12 rather than the expected fd42:b545:2e58:ec06::11 which is causing connection resets.
It looks like it is picking the first entry with the same target address for SNAT rather than also considering the target port too.
This is happening on both IPv4 and IPv6.
A more recent version of OVN will be tried and if the problem still exists will be reported to the upstream project.
