I have been trying to add a remote server to an OVN network, I think I am very close, but just cannot seem to make it work.
So I have an Incus installation on 10.0.1.30 which has a configured ovn network (ovn1, 10.137.5.0/24). I have exposed the northbridge and southbridge server ports, and for now all encryption is disabled.
I wish to add the ‘remote’ server 10.0.1.31 to that network with the address 10.137.5.100.
(This is a proof of concept, in reality the remote server will be connected via a relatively low bandwidth/higher latency link, so it is not suitable to create a cluster.)
I have set /etc/defaults/ovn-central on the remote server to:
OVN_CTL_OPTS="--ovn-northd-sb-db=tcp:10.0.1.30:6642 --ovn-northd-nb-db=tcp:10.0.1.30:6641"
On the remote I have connected the switch using:
ovs-vsctl set open_vswitch . external_ids:ovn-remote=tcp:10.0.1.30:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=10.0.1.31
The ‘chassis’ shows up on the Incus server:
Chassis "04ce9ab0-0249-424b-88c6-463407f20b83"
hostname: incus1
Encap geneve
ip: "127.0.0.1"
options: {csum="true"}
Port_Binding cr-incus-net6-lr-lrp-ext
Port_Binding incus-net6-instance-099efabb-6a5d-40a2-857e-c68f5bfe169a-eth0
Chassis "eaa90051-5b84-4ad6-befc-660927a9e205"
hostname: remote
Encap geneve
ip: "10.0.1.31"
options: {csum="true"}
On the Incus server I have created a new port on the internal switch:
ovn-nbctl lsp-add incus-net6-ls-int extprt
Then on the remote I configure an interface:
ovs-vsctl add-port br-int extprt -- set interface extprt type=internal
ip link set extprt up
ovs-vsctl set interface extprt external_ids:iface-id=extprt
After doing this (and possibly running ovn-sbctl lsp-bind extprt eaa90051-5b84-4ad6-befc-660927a9e205 on the Incus server, not sure if it made a difference), the ovn-sbctl show command shows:
Chassis "04ce9ab0-0249-424b-88c6-463407f20b83"
hostname: incus1
Encap geneve
ip: "127.0.0.1"
options: {csum="true"}
Port_Binding cr-incus-net6-lr-lrp-ext
Port_Binding incus-net6-instance-099efabb-6a5d-40a2-857e-c68f5bfe169a-eth0
Chassis "eaa90051-5b84-4ad6-befc-660927a9e205"
hostname: incus2
Encap geneve
ip: "10.0.1.31"
options: {csum="true"}
Port_Binding extprt
(With Port_Binding extprt now set).
ovn-nbctl lsp-get-up extprt on the Incus service returns ‘up’.
I have then set the ip address of exprt on the remote to 10.137.5.100/24 and then from a container on the ovn1 network I attempt to point that address.
I have attempted to set the address on the server end using ovn-nbctl lsp-set-addresses extprt "1e:5d:bc:ed:52:6f static:10.137.5.100" (with various mac addresses for the br-int, through to the extprt interface).
I can see traffic appearing on the remote using tcpdump, however in /var/log/openvswitch/ovs-vswitchd.log I get:
2025-05-22T12:47:02.808Z|00090|ofproto_dpif_upcall(handler6)|INFO|received packet on unassociated datapath port 3 (no OpenFlow tunnel port for this packet)
2025-05-22T13:08:04.423Z|00091|tunnel(handler6)|WARN|Dropped 19 log messages in last 1262 seconds (most recently, 1252 seconds ago) due to excessive rate
2025-05-22T13:08:04.423Z|00092|tunnel(handler6)|WARN|receive tunnel port not found (icmp6,tun_id=0x3,tun_src=10.0.1.30,tun_dst=10.0.1.31,tun_ipv6_src=::,tun_ipv6_dst=::,tun_gbp_id=0,tun_gbp_flags=0,tun_tos=0,tun_ttl=64,tun_erspan_ver=0,gtpu_flags=0,gtpu_msgtype=0,tun_flags=csum|key,in_port=3,vlan_tci=0x0000,dl_src=10:66:6a:cb:03:81,dl_dst=33:33:00:00:00:02,ipv6_src=fe80::1266:6aff:fecb:381,ipv6_dst=ff02::2,ipv6_label=0x695b5,nw_tos=0,nw_ecn=0,nw_ttl=255,nw_frag=no,icmp_type=133,icmp_code=0)
2025-05-22T13:08:04.423Z|00093|ofproto_dpif_upcall(handler6)|INFO|Dropped 9 log messages in last 1261 seconds (most recently, 1253 seconds ago) due to excessive rate
2025-05-22T13:08:04.423Z|00094|ofproto_dpif_upcall(handler6)|INFO|received packet on unassociated datapath port 3 (no OpenFlow tunnel port for this packet)
If I throw individual packets from the Incus instance on 10.137.5.2 to 10.137.5.100 I can see the system attempting to find the endpoint:
14:12:20.887233 IP 10.0.1.30.10023 > 10.0.1.31.6081: Geneve, Flags [C], vni 0x3, options [8 bytes]: ARP, Request who-has 10.137.5.100 tell 10.137.5.2, length 28
But its not coming together!
Does anyone have any thoughts? I have not been able to find examples of anyone doing the same - but I could have easily missed something.
Many thanks,
- Martyn