Hi, I finally managed to get a reproducible deployment for my cluster including OVN and an IC running over a TINC trunk, seemed to work quite well. (So cluster behind a NAT firewall [homelab] connected via IC to standalone Incus running as a gateway on Digital Ocean)
Only thing left to do seemed to be to add SSL … none of the Incus examples seem to mention this too much, but I think I managed to cobble together all the required settings.
After much experimenting, I finally now have my OVN and OVN-IC running, all the peers are up, but I’ve come unstuck somewhere.
ovn-*ctl, ovn-ic-*ctl now all hang in exactly the same way, despite the Raft peering all looking happy. For example;
ovn-ic-nbctl --db=ssl:192.168.234.10:6647 -p/etc/ovn/key.pem -c/etc/ovn/cert.pem -C/etc/ovn/ca.pem show
Hangs, and all I see in /var/log/ovn/ovsdb-server-ic-nb.log is;
raft|INFO|ssl:192.168.234.10:47684: ovsdb error: expecting notify RPC but received request
So I know the fundamental setup works fine without SSL.
The Raft stuff is all happy with the SSL.
The certificates and port number are all correct because otherwise that would be logging.
… it just hangs.
* Cluster status for core (north) : - ✔ :core: ovs-appctl -t /run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
* Cluster status for core (south) : - ✔ :core: ovs-appctl -t /run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound
* Cluster status for core (ic-north) : - ✔ :core: ovs-appctl -t /run/ovn/ovn_ic_nb_db.ctl cluster/status OVN_IC_Northbound
* Cluster status for core (ic-south) : - ✔ :core: ovs-appctl -t /run/ovn/ovn_ic_sb_db.ctl cluster/status OVN_IC_Southbound
Defined Status for Zone AZ_LOCAL
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
┃ Node Name ┃ North ┃ South ┃ IC North ┃ IC South ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
│ core │ b069 LEADER │ 2aef LEADER │ 14d5 LEADER │ cdd0 LEADER │
│ grok │ 2d71 Member 127 ms │ 6f76 Member 307 ms │ 138d Member 281 ms │ 5cc0 Member 239 ms │
│ p400 │ cefe Member 126 ms │ e3f4 Member 4713396 ms │ 0940 Member 280 ms │ 96fe Member 238 ms │
└───────────┴────────────────────┴────────────────────────┴────────────────────┴────────────────────┘
Does anyone have any ideas as to what could cause ovn-nbctl et al to just hang with an “expecting notify” error … or is there anything obviously wrong with the config? I’m at a bit of a loss as to what to try next …
(this initially showed up as “incus network list” hanging, but tracking it back the underlying access to the database via SSL seems to be causing the incus hang)
/etc/default/ovn-central
VN_CTL_OPTS=" \
--db-nb-cluster-local-addr=192.168.2.1\
--db-sb-cluster-local-addr=192.168.2.1 \
--db-nb-cluster-local-proto=ssl \
--db-sb-cluster-local-proto=ssl \
--db-nb-cluster-remote-proto=ssl \
--db-sb-cluster-remote-proto=ssl \
--ovn-northd-nb-db=ssl:192.168.2.1:6643,ssl:192.168.2.3:6643,ssl:192.168.2.4:6643 \
--ovn-northd-sb-db=ssl:192.168.2.1:6644,ssl:192.168.2.3:6644,ssl:192.168.2.4:6644 \
--ovn-controller-ssl-key=/etc/ovn/key.pem \
--ovn-controller-ssl-cert=/etc/ovn/cert.pem \
--ovn-controller-ssl-ca-cert=/etc/ovn/ca.pem \
--ovn-northd-ssl-key=/etc/ovn/key.pem \
--ovn-northd-ssl-cert=/etc/ovn/cert.pem \
--ovn-northd-ssl-ca-cert=/etc/ovn/ca.pem \
--ovn-nb-db-ssl-key=/etc/ovn/key.pem \
--ovn-nb-db-ssl-cert=/etc/ovn/cert.pem \
--ovn-nb-db-ssl-ca-cert=/etc/ovn/ca.pem \
--ovn-sb-db-ssl-key=/etc/ovn/key.pem \
--ovn-sb-db-ssl-cert=/etc/ovn/cert.pem \
--ovn-sb-db-ssl-ca-cert=/etc/ovn/ca.pem \
--ovn-ic-ssl-key=/etc/ovn/key.pem \
--ovn-ic-ssl-cert=/etc/ovn/cert.pem \
--ovn-ic-ssl-ca-cert=/etc/ovn/ca.pem \
--ovn-ic-nb-db-ssl-key=/etc/ovn/key.pem \
--ovn-ic-nb-db-ssl-cert=/etc/ovn/cert.pem \
--ovn-ic-nb-db-ssl-ca-cert=/etc/ovn/ca.pem
--ovn-ic-sb-db-ssl-key=/etc/ovn/key.pem \
--ovn-ic-sb-db-ssl-cert=/etc/ovn/cert.pem \
--ovn-ic-sb-db-ssl-ca-cert=/etc/ovn/ca.pem \
"
/etc/ovn/ovn-ic-db-params.conf
--ovnnb-db=ssl:192.168.2.1:6643,ssl:192.168.2.3:6643,ssl:192.168.2.4:6643
--ovnsb-db=ssl:192.168.2.1:6644,ssl:192.168.2.3:6644,ssl:192.168.2.4:6644
--ic-nb-db=ssl:192.168.234.10:6647,ssl:192.168.234.12:6647,ssl:192.168.234.14:6647
--ic-sb-db=ssl:192.168.234.10:6648,ssl:192.168.234.12:6648,ssl:192.168.234.14:6648
--private-key=/etc/ovn/key.pem
--certificate=/etc/ovn/cert.pem
--ca-cert=/etc/ovn/ca.pem
/etc/default/ovn-ic
OVN_CTL_OPTS=" \
--db-ic-nb-addr=192.168.234.10 \
--db-ic-sb-addr=192.168.234.10 \
--db-ic-nb-cluster-local-addr=192.168.234.10 \
--db-ic-sb-cluster-local-addr=192.168.234.10 \
--db-ic-nb-cluster-local-proto=ssl \
--db-ic-sb-cluster-local-proto=ssl \
--ovn-northd-nb-db=ssl:192.168.2.1:6643,ssl:192.168.2.3:6643,ssl:192.168.2.4:6643 \
--ovn-northd-sb-db=ssl:192.168.2.1:6644,ssl:192.168.2.3:6644,ssl:192.168.2.4:6644 \
--db-ic-nb-cluster-remote-proto=ssl \
--db-ic-sb-cluster-remote-proto=ssl \
--ovn-ic-ssl-key=/etc/ovn/key.pem \
--ovn-ic-ssl-cert=/etc/ovn/cert.pem \
--ovn-ic-ssl-ca-cert=/etc/ovn/ca.pem \
--ovn-ic-nb-db-ssl-key=/etc/ovn/key.pem \
--ovn-ic-nb-db-ssl-cert=/etc/ovn/cert.pem \
--ovn-ic-nb-db-ssl-ca-cert=/etc/ovn/ca.pem
--ovn-ic-sb-db-ssl-key=/etc/ovn/key.pem \
--ovn-ic-sb-db-ssl-cert=/etc/ovn/cert.pem \
--ovn-ic-sb-db-ssl-ca-cert=/etc/ovn/ca.pem \
--ovn-northd-ssl-key=/etc/ovn/key.pem \
--ovn-northd-ssl-cert=/etc/ovn/cert.pem \
--ovn-northd-ssl-ca-cert=/etc/ovn/ca.pem \
"