I want to connect a few Incus cluster located at different locations and share a few services. Looking at the different network options it boils down to
using VXLAN on top of a VPN
Using OVN with remote network peering on top of VPN
I had success to setup unicast VXLAN using Wireguard to secure the communication and it works pretty well. But it comes with a few drawbacks which can be solved.
Reviewing the documentation for OVN and remote network peering it should work in a similar way but I’m not sure how to configure it correctly. Is it actually possible with the current implementation? As the announcement and documentation isn’t 100% clear about it.
Yeah, OVN interconnection peering should work fine for this assuming you have working OVN on both sides.
It’s not the easiest thing to set up, but also not the hardest, just the usual lack of OVN documentation being the issue
If you’re using TLS for your OVN deployments, you’ll first need to make sure that they’re both using the same CA as OVN interconnection currently requires that the certificates of the interconnection DBs and that of the OVN cluster be the same. As both clusters need to connect to the interconnection DBs, it effectively requires all clusters use the same CA.
Then you need to setup the extra OVN bits, so OVN IC-NB, OVN IC-SB, OVN IC processes themselves in both clusters, make sure both OVN clusters have a name defined and then mark the individual OVS instances as IC chassis.
Once that’s all done, you can add a network integration in each cluster, providing the connection details to your IC NB and SB and after that you can finally peer networks on each cluster.
Do note that this is all L3 peering, so the network on each cluster will have its own subnet and just be able to directly route to the equivalent network on the other cluster all without leaving OVN.
Thanks for the useful input @stgraber.
Agree that the documentation for interconnection lacks quite some detail or useful examples I would say. My research around this topic didn’t brought up much, the only helpful one was this Hands-on with OVN Interconnection - Andreas Karis Blog.
One of the unclear things have been where does the core OVN setup stop and Incus takes over I have some experience in setup a OVN cluster and all the nity greeting pitfalls you can run into, so I was prepared to have similar issues with interconnection.
During the last days I tried to make things work but haven’t been successful. Following the provided Ansible example, reading the docs multiple times and with the help of the link above I was able to get the OVN part working at least as far as I can tell.
In my simple setup I have two nodes let’s call them “tom” and “jerry”. Each of them has a single node OVN setup. I configured all the required services of OVN and “ovn-sbctl show” list the chassis from both nodes. Created a IC-DB on tom and started the OVN-IC service on both nodes and configured the gateways. “OVN-IC-SBCTL” list both zones “az-tom” and “az-jerry” against the IC-DB.
On Incus I created an OVN network on both nodes “ovn-tom” and “ovn-jerry”, deployed an instance like in Incus docs. No real hickups, they can ping the world and internal local IP’s.
Now the fun part starts where I properly miss a step or two. On each node create a “network integration” and “network peering”
Node “tom”:
As only “tom” has the IC-DB both have to use “tom-ip” otherwise it won’t work. Regardless how I configured the peering or integration a ping to any IP of the other network failed.
As mentioned there is properly a step missing or the configuration is simply somewhere wrong.
From OVN cluster network peering experience you have to create a peering from network “ovn-a” to “ovn-b” and the same from network “ovn-b” to “ovn-a”. Without doing this the peering status will show failed. In this case it is always created not much of help. What I’m kind of missing is where to define the remote network for remote peering?
That should then result in a shared transit switch name and in the two getting peered.
Note that you can also switch how the transit switch is named to add/remove specific fields through ovn.transit.pattern. For example adding the peerName to the name then allows you to have multiple peers on the same network, setting up multiple ECMP paths between the networks.
Right, makes sense if you think about it. We are on network level and having two switch without any connection won’t work.
Changed it the way you suggested and still no joy. To make sure I haven’t messed something up I recreated the whole setup a few times but didn’t have any luck at all
Although there is now only one switch it feels there is still something missing.
root@tom:~# ovn-ic-sbctl show
availability-zone jerry
gateway 9e883d2e-bace-4cc9-8a31-41bc75ecb672
hostname: jerry
type: geneve
ip: 10.1.89.167
port ts-incus-ovn-region-default-ovn-ic-jerry
transit switch: ts-incus-ovn-region-default-ovn-ic
address: ["00:16:3e:0f:9a:84 169.254.50.66/28 fd42:f0c5:656:8321::2/64"]
availability-zone tom
gateway fe49c512-e599-4c52-af97-5b637d820495
hostname: tom
type: geneve
ip: 10.1.89.39
port ts-incus-ovn-region-default-ovn-ic-tom
transit switch: ts-incus-ovn-region-default-ovn-ic
address: ["00:16:3e:bd:8c:53 169.254.50.65/28 fd42:f0c5:656:8321::1/64"]
To keep it simple and re-produceable I created an Install script which does the full install / configuration, see link.
It just requires to VM’s tom and jerry to be created, push and execute the script on each node…
Okay, that looks promising, the ovn-ic-sbctl show output looks exactly as I would expect it.
Now let’s see what that looks like on the ovn-nbctl show side for both clusters.
Additionally, use incus network info to get the Logical router name on both cluster, then run ovn-nbctl lr-route-list LRNAME.
Basically we should see a port in the logical switch of each network going into the transit switch and we should see routes for the remote network show up in the logical router routing table.
root@tom:~# incus network ls
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| NAME | TYPE | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY | STATE |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| br-int | bridge | NO | | | | 0 | |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| enp5s0 | physical | NO | | | | 0 | |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| genev_sys_6081 | unknown | NO | | | | 0 | |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| incusbr0 | bridge | YES | 10.1.1.1/24 | none | | 2 | CREATED |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| incusovn1 | bridge | NO | | | | 0 | |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| incusovn1a | unknown | NO | | | | 0 | |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| incusovn1b | unknown | NO | | | | 0 | |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| lo | loopback | NO | | | | 0 | |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| ovn-ic | ovn | YES | 10.1.2.1/24 | none | | 1 | CREATED |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| ovs-system | unknown | NO | | | | 0 | |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
root@tom:~# ovn-sbctl show
Chassis "61b41d6f-81bd-42ad-9f1b-032ce0945586"
hostname: tom
Encap geneve
ip: "10.102.89.110"
options: {csum="true"}
Port_Binding incus-net2-instance-5953435b-94e7-4f77-8533-c8cdf3ef614f-eth0
Port_Binding cr-incus-net2-lr-lrp-ext
Port_Binding cr-ts-incus-ovn-region-default-ovn-ic
Chassis "0a1c90a3-1d9b-4221-8c0b-b2bfbb45fa11"
hostname: jerry
Encap geneve
ip: "10.102.89.41"
options: {csum="true"}
Port_Binding ts-incus-ovn-region-default-ovn-ic-jerry
root@tom:~# ovn-nbctl show
switch 31f2a1e5-75a7-4b72-b08e-dce40169439d (incus-net2-ls-ext)
port incus-net2-ls-ext-lsp-router
type: router
router-port: incus-net2-lr-lrp-ext
port incus-net2-ls-ext-lsp-provider
type: localnet
addresses: ["unknown"]
switch a4ba2775-a982-4e33-88f3-c4d829ceb669 (incus-net2-ls-int)
port incus-net2-instance-5953435b-94e7-4f77-8533-c8cdf3ef614f-eth0
addresses: ["00:16:3e:9a:25:83 dynamic"]
port incus-net2-ls-int-lsp-router
type: router
router-port: incus-net2-lr-lrp-int
switch d0a7d1fb-0391-4be9-ae75-0f724fd23f2a (ts-incus-ovn-region-default-ovn-ic)
port ts-incus-ovn-region-default-ovn-ic-jerry
type: remote
addresses: ["00:16:3e:5e:bc:8d 169.254.88.194/28 fd42:4362:a79:80a8::2/64"]
port ts-incus-ovn-region-default-ovn-ic-tom
type: router
router-port: ts-incus-ovn-region-default-ovn-ic
router 55d635bf-4896-4691-8493-732816ffdb10 (incus-net2-lr)
port incus-net2-lr-lrp-ext
mac: "00:16:3e:ea:2d:3e"
networks: ["10.1.1.5/24"]
port incus-net2-lr-lrp-int
mac: "00:16:3e:ea:2d:3e"
networks: ["10.1.2.1/24"]
port ts-incus-ovn-region-default-ovn-ic
mac: "00:16:3e:ea:2d:3e"
networks: ["169.254.88.193/28", "fd42:4362:a79:80a8::1/64"]
nat a1afcb17-284b-474f-9145-d2391301e736
external ip: "10.1.1.5"
logical ip: "10.1.2.0/24"
type: "snat"
root@tom:~# incus network info ovn-ic
Name: ovn-ic
MAC address: 00:16:3e:ea:2d:3e
MTU: 1442
State: up
Type: broadcast
IP addresses:
inet 10.1.2.1/24 (link)
Network usage:
Bytes received: 0B
Bytes sent: 0B
Packets received: 0
Packets sent: 0
OVN:
Chassis: tom
Logical router: incus-net2-lr
root@tom:~# ovn-nbctl lr-route-list incus-net2-lr
IPv4 Routes
Route Table <main>:
0.0.0.0/0 10.1.1.1 dst-ip incus-net2-lr-lrp-ext
Now here is “jerry” view:
root@jerry:~# incus network ls
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| NAME | TYPE | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY | STATE |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| br-int | bridge | NO | | | | 0 | |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| enp5s0 | physical | NO | | | | 0 | |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| genev_sys_6081 | unknown | NO | | | | 0 | |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| incusbr0 | bridge | YES | 10.2.1.1/24 | none | | 2 | CREATED |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| incusovn1 | bridge | NO | | | | 0 | |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| incusovn1a | unknown | NO | | | | 0 | |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| incusovn1b | unknown | NO | | | | 0 | |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| lo | loopback | NO | | | | 0 | |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| ovn-ic | ovn | YES | 10.2.2.1/24 | none | | 1 | CREATED |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
| ovs-system | unknown | NO | | | | 0 | |
+----------------+----------+---------+-------------+------+-------------+---------+---------+
root@jerry:~# ovn-sbctl show
Chassis "0a1c90a3-1d9b-4221-8c0b-b2bfbb45fa11"
hostname: jerry
Encap geneve
ip: "10.102.89.41"
options: {csum="true"}
Port_Binding incus-net2-instance-455f260b-1855-406e-8aee-f731403d00fc-eth0
Port_Binding cr-ts-incus-ovn-region-default-ovn-ic
Port_Binding cr-incus-net2-lr-lrp-ext
Chassis "61b41d6f-81bd-42ad-9f1b-032ce0945586"
hostname: tom
Encap geneve
ip: "10.102.89.110"
options: {csum="true"}
Port_Binding ts-incus-ovn-region-default-ovn-ic-tom
root@jerry:~# ovn-nbctl show
switch 2fac9bf1-35ba-4890-a75e-08b39c452b9f (incus-net2-ls-int)
port incus-net2-instance-455f260b-1855-406e-8aee-f731403d00fc-eth0
addresses: ["00:16:3e:b2:8d:c7 dynamic"]
port incus-net2-ls-int-lsp-router
type: router
router-port: incus-net2-lr-lrp-int
switch c3d384d2-7ea3-46bd-b1ca-c316b3a95e67 (incus-net2-ls-ext)
port incus-net2-ls-ext-lsp-provider
type: localnet
addresses: ["unknown"]
port incus-net2-ls-ext-lsp-router
type: router
router-port: incus-net2-lr-lrp-ext
switch b42de947-8a8c-42a7-9d1f-b5ef743bd3bf (ts-incus-ovn-region-default-ovn-ic)
port ts-incus-ovn-region-default-ovn-ic-tom
type: remote
addresses: ["00:16:3e:ea:2d:3e 169.254.88.193/28 fd42:4362:a79:80a8::1/64"]
port ts-incus-ovn-region-default-ovn-ic-jerry
type: router
router-port: ts-incus-ovn-region-default-ovn-ic
router 454817e8-8488-41b1-9dc1-38060bde5ee8 (incus-net2-lr)
port ts-incus-ovn-region-default-ovn-ic
mac: "00:16:3e:5e:bc:8d"
networks: ["169.254.88.194/28", "fd42:4362:a79:80a8::2/64"]
port incus-net2-lr-lrp-int
mac: "00:16:3e:5e:bc:8d"
networks: ["10.2.2.1/24"]
port incus-net2-lr-lrp-ext
mac: "00:16:3e:5e:bc:8d"
networks: ["10.2.1.5/24"]
nat f5327eca-9b22-4e45-b72f-96009574ba66
external ip: "10.2.1.5"
logical ip: "10.2.2.0/24"
type: "snat"
root@jerry:~# incus network info ovn-ic
Name: ovn-ic
MAC address: 00:16:3e:5e:bc:8d
MTU: 1442
State: up
Type: broadcast
IP addresses:
inet 10.2.2.1/24 (link)
Network usage:
Bytes received: 0B
Bytes sent: 0B
Packets received: 0
Packets sent: 0
OVN:
Chassis: jerry
Logical router: incus-net2-lr
root@jerry:~# ovn-nbctl lr-route-list incus-net2-lr
IPv4 Routes
Route Table <main>:
0.0.0.0/0 10.2.1.1 dst-ip incus-net2-lr-lrp-ext
Reading the details it seems like all networks and their routing are there? Both sides have the networks listed but a ping between the network still fails…
Btw. did you look at the script (link) I added if there is anthing poping up that would explain this?
The thing that jumps out is that I’m not seeing any OVN configuration for route advertisement, without that, the networks wouldn’t be able to talk to each other without you loading manual routes into OVN.
Try this on both sides and see if that solves it:
ovn-nbctl set NB_Global . options:ic-route-adv=true options:ic-route-learn=true
this was the missing piece of information. Thought that I copied it from the Ansible example you linked above. So sending the exact steps and having a second pair of eyes reviewing was definitely the right approach.
Now tom can ping jerry and the other way around:
root@jerry:~# incus shell ovn1
ovn1:~# ping 10.1.2.2
PING 10.1.2.2 (10.1.2.2): 56 data bytes
64 bytes from 10.1.2.2: seq=0 ttl=62 time=1.687 ms
64 bytes from 10.1.2.2: seq=1 ttl=62 time=0.246 ms
64 bytes from 10.1.2.2: seq=2 ttl=62 time=0.290 ms
64 bytes from 10.1.2.2: seq=3 ttl=62 time=0.370 ms
^C
--- 10.1.2.2 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.246/0.648/1.687 ms
ovn1:~# ping 10.1.2.1
PING 10.1.2.1 (10.1.2.1): 56 data bytes
64 bytes from 10.1.2.1: seq=0 ttl=253 time=0.810 ms
64 bytes from 10.1.2.1: seq=1 ttl=253 time=0.976 ms
^C
--- 10.1.2.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.810/0.893/0.976 ms
ovn1:~# ping 10.1.1.5
PING 10.1.1.5 (10.1.1.5): 56 data bytes
64 bytes from 10.1.1.5: seq=0 ttl=253 time=0.586 ms
64 bytes from 10.1.1.5: seq=1 ttl=253 time=0.613 ms
64 bytes from 10.1.1.5: seq=2 ttl=253 time=0.542 ms
^C
--- 10.1.1.5 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.542/0.580/0.613 ms
ovn1:~#
root@tom:~# incus shell ovn1
ovn1:~# ping 10.2.2.2
PING 10.2.2.2 (10.2.2.2): 56 data bytes
64 bytes from 10.2.2.2: seq=0 ttl=62 time=1.233 ms
64 bytes from 10.2.2.2: seq=1 ttl=62 time=0.338 ms
^C
--- 10.2.2.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.338/0.785/1.233 ms
ovn1:~# ping 10.2.2.1
PING 10.2.2.1 (10.2.2.1): 56 data bytes
64 bytes from 10.2.2.1: seq=0 ttl=253 time=0.716 ms
64 bytes from 10.2.2.1: seq=1 ttl=253 time=0.525 ms
^C
--- 10.2.2.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.525/0.620/0.716 ms
ovn1:~# ping 10.2.1.5
PING 10.2.1.5 (10.2.1.5): 56 data bytes
64 bytes from 10.2.1.5: seq=0 ttl=253 time=0.880 ms
64 bytes from 10.2.1.5: seq=1 ttl=253 time=0.459 ms
^C
--- 10.2.1.5 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.459/0.669/0.880 ms
ovn1:~#
That makes my day!!!
Now it only requires to fix some of the Ansible bugs I discovered (but that is a different story) and get it deployed.
Really appreciate your help and all the hard work you put into the project.
