OVN high availability cluster tutorial

This tutorial describes how to setup a 3 node OVN and LXD high availability cluster.

I’m doing this with 3 LXD VMs connected to a private bridge with subnet 10.98.30.0/24, so lets create them first:

lxc init images:ubuntu/focal v1 --vm
lxc init images:ubuntu/focal v2 --vm
lxc init images:ubuntu/focal v3 --vm

We want to ensure they have statically assigned IPs:

lxc config device override v1 eth0 ipv4.address=10.98.30.2
lxc config device override v2 eth0 ipv4.address=10.98.30.3
lxc config device override v3 eth0 ipv4.address=10.98.30.4
lxc start v1 v2 v3

Install ovn-central in the first VM:

lxc shell v1
sudo apt install ovn-central -y

Update /etc/default/ovn-central with:

OVN_CTL_OPTS= \
  --db-nb-addr=10.98.30.2 \
  --db-sb-addr=10.98.30.2 \
  --db-nb-cluster-local-addr=10.98.30.2 \
  --db-sb-cluster-local-addr=10.98.30.2 \
  --db-nb-create-insecure-remote=yes \
  --db-sb-create-insecure-remote=yes \
  --ovn-northd-nb-db=tcp:10.98.30.2:6641,tcp:10.98.30.3:6641,tcp:10.98.30.4:6641 \
  --ovn-northd-sb-db=tcp:10.98.30.2:6642,tcp:10.98.30.3:6642,tcp:10.98.30.4:6642

Clear existing config, restart ovn-central and expose the DBs to the network:

rm -rvf /var/lib/ovn
systemctl restart ovn-central
ovn-nbctl show
exit

Now install ovn-central on the 2nd and 3rd VMs:

lxc shell v{n}
sudo apt install ovn-central -y

On each VM update /etc/default/ovn-central with the the .n parts changed to the VM’s IP and add the db-nb-cluster-remote-addr and db-sb-cluster-remote-addr settings in order to allow OVN to connect to the first VM:

OVN_CTL_OPTS= \
  --db-nb-addr=10.98.30.n \
  --db-sb-addr=10.98.30.n \
  --db-nb-cluster-local-addr=10.98.30.n \
  --db-sb-cluster-local-addr=10.98.30.n \
  --db-nb-create-insecure-remote=yes \
  --db-sb-create-insecure-remote=yes \
  --ovn-northd-nb-db=tcp:10.98.30.2:6641,tcp:10.98.30.3:6641,tcp:10.98.30.4:6641 \
  --ovn-northd-sb-db=tcp:10.98.30.2:6642,tcp:10.98.30.3:6642,tcp:10.98.30.4:6642 \
  --db-nb-cluster-remote-addr=10.98.30.2 \
  --db-sb-cluster-remote-addr=10.98.30.2  

Clear existing config, restart ovn-central:

rm -rvf /var/lib/ovn
systemctl restart ovn-central

Look in /var/log/ovn/ovn-northd.log for a line like This ovn-northd instance is now on standby.

On each VM check that the databases are working using (you should get empty output but no errors):

OVN_NB_DB=tcp:10.98.30.2:6641,tcp:10.98.30.3:6641,tcp:10.98.30.4:6641 ovn-nbctl show
OVN_SB_DB=tcp:10.98.30.2:6642,tcp:10.98.30.3:6642,tcp:10.98.30.4:6642 ovn-sbctl show

Now install ovn-host on each VM and configure OVS to connect to OVN:

sudo apt install ovn-host -y
sudo ovs-vsctl set open_vswitch . \
    external_ids:ovn-encap-type=geneve \
    external_ids:ovn-remote="unix:/var/run/ovn/ovnsb_db.sock" \
    external_ids:ovn-encap-ip=$(ip r get 10.98.30.1 | grep -v cache | awk '{print $5}')

Check that br-int interface exists in output of ip l on each VM.

Now install LXD on each VM and setup a LXD cluster as normal:

sudo apt install snapd -y
sudo snap install lxd
sudo lxd init

Once the LXD cluster is setup, inform LXD how to connect to OVN:

lxc config set network.ovn.northbound_connection=tcp:10.98.30.2:6641,tcp:10.98.30.3:6641,tcp:10.98.30.4:6641

Now lets create a bridged network for use as an OVN uplink network (you can also use a dedicated spare physical interface for uplinking onto an external network, see Networks | LXD):

lxc network create lxdbr0 --target=v1
lxc network create lxdbr0 --target=v2
lxc network create lxdbr0 --target=v3
lxc network create lxdbr0 \
	ipv4.address=10.179.176.1/24 \
    ipv4.nat=true \
	ipv4.dhcp.ranges=10.179.176.5-10.179.176.10 \ # Required to specify ipv4.ovn.ranges
	ipv4.ovn.ranges=10.179.176.11-10.179.176.20 # For use with OVN network's router IP on the uplink network

Now lets create an OVN network using the lxdbr0 bridge as an uplink:

lxc network create ovn0 --type=ovn network=lxdbr0

Finally lets check we can create an instance that connects to our OVN network:

lxc shell v3
lxc init images:ubuntu/focal c1
lxc config device add c1 eth0 nic network=ovn0
lxc start c1
lxc ls
+------+---------+-------------------+-----------------------------------------------+-----------+-----------+----------+
| NAME |  STATE  |       IPV4        |                     IPV6                      |   TYPE    | SNAPSHOTS | LOCATION |
+------+---------+-------------------+-----------------------------------------------+-----------+-----------+----------+
| c1   | RUNNING | 10.34.84.2 (eth0) | fd42:a21f:aa90:7cd7:216:3eff:fe7d:3b8a (eth0) | CONTAINER | 0         | v1       |
+------+---------+-------------------+-----------------------------------------------+-----------+-----------+----------+
lxc exec c1 -- ping 8.8.8.8

For more info about OVN network settings see:

https://linuxcontainers.org/lxd/docs/master/networks#network-ovn

2 Likes

What is the relationship between this clustering style and fan networking? If I recall correctly, there is a prompt in lxd init regarding setting up a fan network overlay.

Fan is the only clustering method I have used, but I’m very interested in this one.

The fan networking is certainly the easiest way to get networking working across a cluster. The main differences between fan and OVN are:

  • fan subnets are host-specific, moving an instance between servers in a cluster will lead to a different IP address. With OVN you get virtual L2 networks across your cluster so can move things around and not change addresses.
  • fan networks are system-wide and must not overlap, so you can’t delegate their creation to untrusted/restricted users of your cluster and running multiple fan networks on the same cluster requires you managing non-conflicting underlay subnets. With OVN, your networks never show up on the host system so you can reuse the same subnet many times if you feel like without ever getting a conflict. The underlay is a set of auto-generated geneve tunnels between your servers so no need to think about underlay subnets. This means untrusted/restricted users can create their own networks in their own project without being able to impact anyone else.

Another difference is that OVN allows for distributed firewalling (through flow rules) which integrates with LXD’s new network ACL feature. This allows very fine grained firewalling even within a network including label based source/destination rules so you don’t need to hardcode addresses everywhere. Traditional Linux networking (including the fan overlay) are quite a bit more limited in that regard and can get very confusing when dealing with cross-host traffic.

The obvious downside is that OVN requires you to have ovn and openvswitch installed and configured on your cluster nodes, similar in a way to what we have on the storage side with ceph. But once it’s in place, it’s very flexible and we have a number of extra features coming soon which will make it an even better option for many users.

2 Likes