Summary
I recently setup a secure OVN cluster and am now trying to have LXD connect to the northbound database. The first issue is that it is looking for the NB database’s x509 certificate, key, and CA in the default locations and I’m not sure if it is possible to tell it to look in a different location. I can copy my certificate, key, and CA to these locations to get around this first problem, but it also throws a database error: database connection failed (Address family not supported by protocol).
Version
$ lxd version
5.13
$ snap list lxd
Name Version Rev Tracking Publisher Notes
lxd 5.13-8e2d7eb 24846 5.13/stable canonical✓ -
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy
$ dpkg -l | grep ovn
ii ovn-central 22.03.2-0ubuntu0.22.04.1 amd64 OVN central components
ii ovn-common 22.03.2-0ubuntu0.22.04.1 amd64 OVN common components
ii ovn-host 22.03.2-0ubuntu0.22.04.1 amd64 OVN host components
Steps
These steps assume that a working OVN cluster has been setup using OVS’s passive SSL connection types instead of the default TCP connection type [1].
First I specified the network.ovn.northbound_connection.
$ lxc config set network.ovn.northbound_connection ssl:sfllxc01-cloud.harmonywave.com:6641,ssl:sfllxc02-cloud.harmonywave.com:6641,ssl:sfllxc03-cloud.harmonywave.com:6641
Note that I’m using ssl instead of tcp for the connection method [1]. Next I created my uplink network which worked successfully.
$ lxc network create lxd-mgmt-uplink --type=physical parent=bond-svc mtu=9000 vlan=28 dns.nameservers="74.82.42.42,1.1.1.1" ipv4.gateway=10.3.28.1/22 ipv4.ovn.ranges="10.3.31.145-10.3.31.192" ipv6.gateway=2620:5:e000:301c::1/64 ipv6.ovn.ranges="2620:5:e000:301c:0:11::-2620:5:e000:301c::11:ffff:ffff"
If I don’t move my cert/key/CA to what looks like ovn-nbctl’s default locations it fails with a No such file or directory and the database connection failed (Address family not supported by protocol) errors.
$ lxc network create ovn-mgmt --type ovn network=lxd-mgmt-uplink bridge.mtu=8942 dns.domain=harmonywave.com dns.search="harmonywave.com,harmonywave.cloud"
Error: Failed to run: ovn-nbctl --timeout=10 --db ssl:sfllxc01-cloud.harmonywave.com:6641,ssl:sfllxc02-cloud.harmonywave.com:6641,ssl:sfllxc03-cloud.harmonywave.com:6641 -c /etc/ovn/cert_host -p /etc/ovn/key_host -C /etc/ovn/ovn-central.crt --wait=sb ha-chassis-group-add lxd-net9: exit status 1 (2023-06-29T14:34:35Z|00001|stream_ssl|ERR|SSL_use_certificate_file: error:80000002:system library::No such file or directory
2023-06-29T14:34:35Z|00002|stream_ssl|ERR|SSL_use_PrivateKey_file: error:10080002:BIO routines::system lib
2023-06-29T14:34:35Z|00003|stream_ssl|ERR|failed to load client certificates from /etc/ovn/ovn-central.crt: error:0A080002:SSL routines::system lib
2023-06-29T14:34:35Z|00005|stream_ssl|ERR|ssl:sfllxc02-cloud.harmonywave.com:6641: connect: Address family not supported by protocol
2023-06-29T14:34:35Z|00006|stream_ssl|ERR|ssl:sfllxc03-cloud.harmonywave.com:6641: connect: Address family not supported by protocol
2023-06-29T14:34:35Z|00007|stream_ssl|ERR|ssl:sfllxc01-cloud.harmonywave.com:6641: connect: Address family not supported by protocol
ovn-nbctl: ssl:sfllxc01-cloud.harmonywave.com:6641,ssl:sfllxc02-cloud.harmonywave.com:6641,ssl:sfllxc03-cloud.harmonywave.com:6641: database connection failed (Address family not supported by protocol))
Is there a way to pass the cert/key/CA locations to LXD so it can use them with the ovn-nbctl command? Even if there isn’t, the bigger problem is the database connection failed error. If I copy the cert/key/CA to these default locations I still get the database error, but I can run the ovn-nbctl command exactly as it’s shown and it works successfully outside of LXD.
# Run the `lxc network create` command again which will show the ovn-nbctl command that was run and the database error that is thrown.
$ lxc network create ovn-mgmt --type ovn network=lxd-mgmt-uplink bridge.mtu=8942 dns.domain=harmonywave.com dns.search="harmonywave.com,harmonywave.cloud"
Error: Failed to run: ovn-nbctl --timeout=10 --db ssl:sfllxc01-cloud.harmonywave.com:6641,ssl:sfllxc02-cloud.harmonywave.com:6641,ssl:sfllxc03-cloud.harmonywave.com:6641 -c /etc/ovn/cert_host -p /etc/ovn/key_host -C /etc/ovn/ovn-central.crt --wait=sb ha-chassis-group-add lxd-net12: exit status 1 (2023-06-29T15:06:53Z|00002|stream_ssl|ERR|ssl:sfllxc01-cloud.harmonywave.com:6641: connect: Address family not supported by protocol
2023-06-29T15:06:53Z|00003|stream_ssl|ERR|ssl:sfllxc02-cloud.harmonywave.com:6641: connect: Address family not supported by protocol
2023-06-29T15:06:53Z|00004|stream_ssl|ERR|ssl:sfllxc03-cloud.harmonywave.com:6641: connect: Address family not supported by protocol
ovn-nbctl: ssl:sfllxc01-cloud.harmonywave.com:6641,ssl:sfllxc02-cloud.harmonywave.com:6641,ssl:sfllxc03-cloud.harmonywave.com:6641: database connection failed (Address family not supported by protocol))
# List any ha chassis groups. There are none because the above command failed.
$ sudo ovn-nbctl --timeout=10 --db ssl:sfllxc01-cloud.harmonywave.com:6641,ssl:sfllxc02-cloud.harmonywave.com:6641,ssl:sfllxc03-cloud.harmonywave.com:6641 -c /etc/ovn/cert_host -p /etc/ovn/key_host -C /etc/ovn/ovn-central.crt --wait=sb ha-chassis-group-list
$
# Manually run the ovn-nbctl ha-chassis-group-add command. I literally copy and pasted this command from the output of the `lxc network create` command. No errors are thrown
$ sudo ovn-nbctl --timeout=10 --db ssl:sfllxc01-cloud.harmonywave.com:6641,ssl:sfllxc02-cloud.harmonywave.com:6641,ssl:sfllxc03-cloud.harmonywave.com:6641 -c /etc/ovn/cert_host -p /etc/ovn/key_host -C /etc/ovn/ovn-central.crt --wait=sb ha-chassis-group-add lxd-net12
$
# List the ha chassis groups again. Now there is the `lxd-net12` group that was just created manually.
$ sudo ovn-nbctl --timeout=10 --db ssl:sfllxc01-cloud.harmonywave.com:6641,ssl:sfllxc02-cloud.harmonywave.com:6641,ssl:sfllxc03-cloud.harmonywave.com:6641 -c /etc/ovn/cert_host -p /etc/ovn/key_host -C /etc/ovn/ovn-central.crt --wait=sb ha-chassis-group-list
987a987a-2986-4b52-91f3-15fcea5c5c91 (lxd-net12)
$
Any hints of what I could be doing wrong?
[1] https://docs.openvswitch.org/en/latest/ref/ovsdb.7/#connection-methods