Is there an advanced ingress video which covers ovn?
I see vids covering network bridges and a single node.
I am looking for something that coves ovn. Discusses setting a shared bridge on the uplink nic or how your supposed to do that. Covers trouble shooting and where to look for issues… eg firewall logging. Or ovn logs. What is the traffic flow for ingress etc?
What might cause the proxy to only work one etc. Does the controller change between chassis etc
We have https://www.youtube.com/watch?v=B-Uzo9WldMs but I expect you’ve seen that already.
We have https://www.youtube.com/watch?v=1M__Rm9iZb8 for general OVN cluster setup (including uplink).
What is the issue you are encountering?
I am struggling with consistent ingress for forwards and proxies. The proxy works once and the forward not at all.
I have setup a standalone bridge on each instance.
I set the external subnet as a route on this dummy bridge.
My test vms have 3 physiacal nics. One for internal ovn traffic. One for ceph traffic and one that is the external interface.
I created a new bridge called uplink. And a new ovn network that uses the uplink bridge as its uplink and uses nat for outbound traffic.
I am trying to simulate the hardware i will request from the isp to make this work.
External nic. /28 14 usable ips but 3 will be used for lxd nodes.
Ovn nic (internal). 10.40.0.x
Ceph nic (internal). 10.41.0.x
Uplink (br) (lxd managed)(no hardware) 10.0.0.x. → lxdbr0(lxd managed)(ovn). 10.1.1.x
Do you have a LXD cluster for OVN or a single LXD server?
Its a cluster.
I setup 3 vms as a cluster.
OK so LXD’s OVN networks require a shared L2 for the uplink network.
See Linux Containers - LXD - Has been moved to Canonical
A high availability OVN cluster requires a shared layer 2 network, so that the active OVN chassis can move between cluster members (which effectively allows the OVN router’s external IP to be reachable from a different host).
Therefore, you must specify either an unmanaged bridge interface or an unused physical interface as the parent for the physical network that is used for OVN uplink.
Using a LXD managed private bridge for the OVN uplink is OK for standalone servers and testing, but no good for clustered setups.
This is because OVN will select a single cluster member to be used for the uplink connection for instances on all the LXD cluster members.
Are you suggesting another internal nic for this traffic?
Yes, or an unmanaged bridge that is connected to the same external network as each of the cluster members.
@tomp I think I am very close now. from my v1 v2 v3 instances I can now connect to a port and see data transfer… however the last piece is not working I cannot see the assigned ip from the host that is hosting v1 v2 v3 … I can see the bridged interfaces
the host bridge does own the entire subnet…
Please show ip a and ip r output from v1, v2 and v3, along with the host you’re testing from.
Also please show lxc network show <uplink> and lxc network show <ovn>
Underlying host physical hardware
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp7s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether bc:ae:c5:62:dd:54 brd ff:ff:ff:ff:ff:ff
inet 192.168.5.6/24 brd 192.168.5.255 scope global noprefixroute enp7s1
valid_lft forever preferred_lft forever
inet6 2001:470:4470::18e/128 scope global dynamic noprefixroute
valid_lft 39131sec preferred_lft 39131sec
inet6 fd23:c1f1:f9c::18e/128 scope global dynamic noprefixroute
valid_lft 39131sec preferred_lft 39131sec
inet6 fd23:c1f1:f9c:0:4694:61e3:c642:5fe/64 scope global temporary dynamic
valid_lft 540297sec preferred_lft 21307sec
inet6 fd23:c1f1:f9c:0:1cfe:7600:ec7d:c078/64 scope global mngtmpaddr noprefixroute
valid_lft forever preferred_lft forever
inet6 2001:470:4470:0:91fc:127f:e2b7:876d/64 scope global temporary dynamic
valid_lft 540297sec preferred_lft 21307sec
inet6 2001:470:4470:0:2e80:bbf2:ea15:6c0c/64 scope global mngtmpaddr noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::75f5:34c3:8e9c:49f7/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:e8:18:0b brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
4: lxcbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 00:16:3e:00:00:00 brd ff:ff:ff:ff:ff:ff
inet 10.0.3.1/24 brd 10.0.3.255 scope global lxcbr0
valid_lft forever preferred_lft forever
5: zthnhhq2ry: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2800 qdisc fq_codel state UNKNOWN group default qlen 1000
link/ether 0e:99:ef:d6:76:74 brd ff:ff:ff:ff:ff:ff
inet 192.168.191.183/24 brd 192.168.191.255 scope global zthnhhq2ry
valid_lft forever preferred_lft forever
inet 192.168.191.186/24 brd 192.168.191.255 scope global secondary zthnhhq2ry
valid_lft forever preferred_lft forever
inet6 fdaf:78bf:9436:c7fe:e99:9367:28e0:e2cb/88 scope global
valid_lft forever preferred_lft forever
inet6 fc99:bf41:9a67:28e0:e2cb::1/40 scope global
valid_lft forever preferred_lft forever
inet6 fe80::c99:efff:fed6:7674/64 scope link
valid_lft forever preferred_lft forever
6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:6f:83:e9:82 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
7: br-0c9840d94d0e: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:3d:51:86:a3 brd ff:ff:ff:ff:ff:ff
inet 172.23.0.1/16 brd 172.23.255.255 scope global br-0c9840d94d0e
valid_lft forever preferred_lft forever
8: br-c45814d16427: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:4d:4e:00:22 brd ff:ff:ff:ff:ff:ff
inet 172.27.0.1/16 brd 172.27.255.255 scope global br-c45814d16427
valid_lft forever preferred_lft forever
9: br-c8bd14a80735: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:ce:5b:12:2e brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 brd 172.18.255.255 scope global br-c8bd14a80735
valid_lft forever preferred_lft forever
10: mpqemubr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:35:84:28 brd ff:ff:ff:ff:ff:ff
inet 10.175.149.1/24 brd 10.175.149.255 scope global mpqemubr0
valid_lft forever preferred_lft forever
11: rh-priv: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 00:16:3e:1c:5c:25 brd ff:ff:ff:ff:ff:ff
inet 10.1.0.1/24 scope global rh-priv
valid_lft forever preferred_lft forever
12: rh-pub: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 00:16:3e:8c:5e:56 brd ff:ff:ff:ff:ff:ff
inet 10.196.125.1/28 scope global rh-pub
valid_lft forever preferred_lft forever
13: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:e5:7c:c5 brd ff:ff:ff:ff:ff:ff
inet 10.96.124.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
inet6 fd42:2c94:34b6:cabc::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fee5:7cc5/64 scope link
valid_lft forever preferred_lft forever
14: lxdbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:e8:41:17 brd ff:ff:ff:ff:ff:ff
inet 10.98.30.1/24 scope global lxdbr1
valid_lft forever preferred_lft forever
inet6 fd42:7386:9481:19b7::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fee8:4117/64 scope link
valid_lft forever preferred_lft forever
15: lxdbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:f6:cf:35 brd ff:ff:ff:ff:ff:ff
inet 10.2.0.1/24 scope global lxdbr2
valid_lft forever preferred_lft forever
16: lxdbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:32:26:bf brd ff:ff:ff:ff:ff:ff
inet 10.3.0.1/24 scope global lxdbr3
valid_lft forever preferred_lft forever
inet6 fd42:8fdd:7c0e:87c1::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe32:26bf/64 scope link
valid_lft forever preferred_lft forever
18: veth2edd9d8e@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 8a:34:65:16:19:e8 brd ff:ff:ff:ff:ff:ff link-netnsid 0
19: tap1f26c73e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr0 state UP group default qlen 1000
link/ether 7e:b1:e1:eb:8b:f7 brd ff:ff:ff:ff:ff:ff
21: vethde5dd531@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 8a:0f:ff:68:e6:8f brd ff:ff:ff:ff:ff:ff link-netnsid 1
22: tap00a8bc43: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr1 state UP group default qlen 1000
link/ether 6e:d4:b7:ad:7d:30 brd ff:ff:ff:ff:ff:ff
23: tap081b9deb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr2 state UP group default qlen 1000
link/ether 8a:b6:9c:ef:a4:4d brd ff:ff:ff:ff:ff:ff
24: tap8fdd3773: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr3 state UP group default qlen 1000
link/ether 86:90:f3:5e:e1:16 brd ff:ff:ff:ff:ff:ff
25: tap59dff295: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr1 state UP group default qlen 1000
link/ether 6a:ef:f7:a3:eb:4f brd ff:ff:ff:ff:ff:ff
26: tap7224ac40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr2 state UP group default qlen 1000
link/ether ba:99:0f:c5:de:45 brd ff:ff:ff:ff:ff:ff
27: tap11c82ce6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr3 state UP group default qlen 1000
link/ether 2e:df:69:1d:82:10 brd ff:ff:ff:ff:ff:ff
28: tap3759d2a8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr1 state UP group default qlen 1000
link/ether c2:3b:cc:b1:89:36 brd ff:ff:ff:ff:ff:ff
29: tapc2d7bc89: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr2 state UP group default qlen 1000
link/ether 96:76:9f:98:11:4e brd ff:ff:ff:ff:ff:ff
30: tap350478e9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr3 state UP group default qlen 1000
link/ether 4a:23:53:d8:42:e5 brd ff:ff:ff:ff:ff:ff
32: veth7511099d@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether c2:3d:2b:62:f4:46 brd ff:ff:ff:ff:ff:ff link-netnsid 2
ip r
default via 192.168.5.1 dev enp7s1 proto static metric 100
10.0.3.0/24 dev lxcbr0 proto kernel scope link src 10.0.3.1 linkdown
10.1.0.0/24 dev rh-priv proto kernel scope link src 10.1.0.1 linkdown
10.2.0.0/24 dev lxdbr2 proto kernel scope link src 10.2.0.1
10.3.0.0/24 dev lxdbr3 proto kernel scope link src 10.3.0.1
10.96.124.0/24 dev lxdbr0 proto kernel scope link src 10.96.124.1
10.98.30.0/24 dev lxdbr1 proto kernel scope link src 10.98.30.1
10.175.149.0/24 dev mpqemubr0 proto kernel scope link src 10.175.149.1 linkdown
10.196.125.0/28 dev rh-pub proto kernel scope link src 10.196.125.1 linkdown
169.254.0.0/16 dev virbr0 scope link metric 1000 linkdown
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br-c8bd14a80735 proto kernel scope link src 172.18.0.1 linkdown
172.23.0.0/16 dev br-0c9840d94d0e proto kernel scope link src 172.23.0.1 linkdown
172.27.0.0/16 dev br-c45814d16427 proto kernel scope link src 172.27.0.1 linkdown
192.168.5.0/24 dev enp7s1 proto kernel scope link src 192.168.5.6 metric 100
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
192.168.191.0/24 dev zthnhhq2ry proto kernel scope link src 192.168.191.183
for vm in v1 v2 v3; do echo $vm;echo ;lxc exec $vm ip a;echo; lxc exec $vm ip r; done
v1
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
link/ether 00:16:3e:58:b9:82 brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br1 state UP group default qlen 1000
link/ether 00:16:3e:a0:29:b0 brd ff:ff:ff:ff:ff:ff
4: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:16:3e:96:bd:bf brd ff:ff:ff:ff:ff:ff
inet 10.3.0.2/24 brd 10.3.0.255 scope global enp7s0
valid_lft forever preferred_lft forever
inet6 fd42:8fdd:7c0e:87c1:216:3eff:fe96:bdbf/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 3282sec preferred_lft 3282sec
inet6 fe80::216:3eff:fe96:bdbf/64 scope link
valid_lft forever preferred_lft forever
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether f6:da:31:98:57:7c brd ff:ff:ff:ff:ff:ff
inet 10.98.30.10/24 metric 100 brd 10.98.30.255 scope global dynamic br0
valid_lft 2806sec preferred_lft 2806sec
inet6 fd42:7386:9481:19b7:f4da:31ff:fe98:577c/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 3174sec preferred_lft 3174sec
inet6 fe80::f4da:31ff:fe98:577c/64 scope link
valid_lft forever preferred_lft forever
6: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 62:4c:bb:3c:cd:b7 brd ff:ff:ff:ff:ff:ff
inet 10.2.0.2/24 brd 10.2.0.255 scope global br1
valid_lft forever preferred_lft forever
inet6 fe80::604c:bbff:fe3c:cdb7/64 scope link
valid_lft forever preferred_lft forever
7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 36:ac:58:2e:66:4e brd ff:ff:ff:ff:ff:ff
9: br-int: <BROADCAST,MULTICAST> mtu 1442 qdisc noop state DOWN group default qlen 1000
link/ether 0a:e4:16:7a:61:1f brd ff:ff:ff:ff:ff:ff
10: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
link/ether 86:2f:f3:46:3a:0a brd ff:ff:ff:ff:ff:ff
inet6 fe80::10cb:e6ff:fe00:2e10/64 scope link
valid_lft forever preferred_lft forever
11: lxdovn2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 06:e5:a6:f1:9f:45 brd ff:ff:ff:ff:ff:ff
12: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:3b:f4:46:f4 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
13: lxdfan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:fd:4c:cf brd ff:ff:ff:ff:ff:ff
inet 240.10.0.1/8 scope global lxdfan0
valid_lft forever preferred_lft forever
16: UPLINK: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:3f:93:c3 brd ff:ff:ff:ff:ff:ff
inet 10.1.0.1/24 scope global UPLINK
valid_lft forever preferred_lft forever
17: lxdovn2b@lxdovn2a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
link/ether 6e:fc:20:37:36:f7 brd ff:ff:ff:ff:ff:ff
18: lxdovn2a@lxdovn2b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master UPLINK state UP group default qlen 1000
link/ether ae:02:46:87:8d:42 brd ff:ff:ff:ff:ff:ff
21: lxdfan0-mtu: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
link/ether 1a:8a:ff:2d:6b:f5 brd ff:ff:ff:ff:ff:ff
inet6 fe80::703f:b3ff:fe70:951/64 scope link
valid_lft forever preferred_lft forever
22: lxdfan0-fan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
link/ether 5a:66:26:d3:66:c4 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5866:26ff:fed3:66c4/64 scope link
valid_lft forever preferred_lft forever
24: veth8324f1a8@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc noqueue master ovs-system state UP group default qlen 1000
link/ether be:e7:8e:aa:f4:d7 brd ff:ff:ff:ff:ff:ff link-netnsid 0
25: lxdovn7b@lxdovn7a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
link/ether c2:73:2a:49:a3:43 brd ff:ff:ff:ff:ff:ff
26: lxdovn7a@lxdovn7b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1 state UP group default qlen 1000
link/ether 7e:8b:06:d2:ac:ff brd ff:ff:ff:ff:ff:ff
27: lxdovn7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 12:84:c9:e9:22:48 brd ff:ff:ff:ff:ff:ff
default via 10.98.30.1 dev br0 proto dhcp src 10.98.30.10 metric 100
10.1.0.0/24 dev UPLINK proto kernel scope link src 10.1.0.1
10.2.0.0/24 dev br1 proto kernel scope link src 10.2.0.2
10.3.0.0/24 dev enp7s0 proto kernel scope link src 10.3.0.2
10.98.30.0/24 via 10.2.0.200 dev br1
10.98.30.0/24 dev br0 proto kernel scope link src 10.98.30.10 metric 100
10.98.30.1 dev br0 proto dhcp scope link src 10.98.30.10 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
240.0.0.0/8 dev lxdfan0 proto kernel scope link src 240.10.0.1
v2
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
link/ether 00:16:3e:02:a8:7f brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br1 state UP group default qlen 1000
link/ether 00:16:3e:cf:c8:e1 brd ff:ff:ff:ff:ff:ff
4: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:16:3e:2f:5b:6a brd ff:ff:ff:ff:ff:ff
inet 10.3.0.3/24 brd 10.3.0.255 scope global enp7s0
valid_lft forever preferred_lft forever
inet6 fd42:8fdd:7c0e:87c1:216:3eff:fe2f:5b6a/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 3281sec preferred_lft 3281sec
inet6 fe80::216:3eff:fe2f:5b6a/64 scope link
valid_lft forever preferred_lft forever
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 0e:d9:5f:19:7d:cd brd ff:ff:ff:ff:ff:ff
inet 10.98.30.148/24 metric 100 brd 10.98.30.255 scope global dynamic br0
valid_lft 3113sec preferred_lft 3113sec
inet6 fd42:7386:9481:19b7:cd9:5fff:fe19:7dcd/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 3174sec preferred_lft 3174sec
inet6 fe80::cd9:5fff:fe19:7dcd/64 scope link
valid_lft forever preferred_lft forever
6: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 7e:f9:a7:8a:9f:29 brd ff:ff:ff:ff:ff:ff
inet 10.2.0.3/24 brd 10.2.0.255 scope global br1
valid_lft forever preferred_lft forever
inet6 fe80::7cf9:a7ff:fe8a:9f29/64 scope link
valid_lft forever preferred_lft forever
7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether d2:47:7d:e1:19:eb brd ff:ff:ff:ff:ff:ff
9: br-int: <BROADCAST,MULTICAST> mtu 1442 qdisc noop state DOWN group default qlen 1000
link/ether 52:9c:3c:01:08:14 brd ff:ff:ff:ff:ff:ff
10: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
link/ether 1e:cc:e7:79:e1:15 brd ff:ff:ff:ff:ff:ff
inet6 fe80::7c74:8aff:fe68:337d/64 scope link
valid_lft forever preferred_lft forever
11: lxdovn2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 92:3f:4d:b8:19:4a brd ff:ff:ff:ff:ff:ff
12: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:a3:3b:ce:eb brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
13: UPLINK: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:3f:93:c3 brd ff:ff:ff:ff:ff:ff
inet 10.1.0.1/24 scope global UPLINK
valid_lft forever preferred_lft forever
14: lxdfan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:4c:47:d5 brd ff:ff:ff:ff:ff:ff
inet 240.148.0.1/8 scope global lxdfan0
valid_lft forever preferred_lft forever
17: lxdovn2b@lxdovn2a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
link/ether 5a:47:91:e1:19:32 brd ff:ff:ff:ff:ff:ff
18: lxdovn2a@lxdovn2b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master UPLINK state UP group default qlen 1000
link/ether 26:4c:b4:b9:e4:0d brd ff:ff:ff:ff:ff:ff
21: lxdfan0-mtu: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
link/ether 02:14:d6:71:58:3b brd ff:ff:ff:ff:ff:ff
inet6 fe80::14:d6ff:fe71:583b/64 scope link
valid_lft forever preferred_lft forever
22: lxdfan0-fan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
link/ether 2e:9a:42:dd:84:ed brd ff:ff:ff:ff:ff:ff
inet6 fe80::2c9a:42ff:fedd:84ed/64 scope link
valid_lft forever preferred_lft forever
24: veth079edabf@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc noqueue master ovs-system state UP group default qlen 1000
link/ether 72:10:a2:df:ea:c0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
27: lxdovn7b@lxdovn7a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
link/ether 62:2f:51:76:3f:4b brd ff:ff:ff:ff:ff:ff
28: lxdovn7a@lxdovn7b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1 state UP group default qlen 1000
link/ether 5e:64:c0:8d:f0:31 brd ff:ff:ff:ff:ff:ff
29: lxdovn7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 3e:4b:6b:be:3e:4c brd ff:ff:ff:ff:ff:ff
33: veth4b08de91@if32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc noqueue master ovs-system state UP group default qlen 1000
link/ether 92:3a:43:5a:6c:c6 brd ff:ff:ff:ff:ff:ff link-netnsid 2
default via 10.98.30.1 dev br0 proto dhcp src 10.98.30.148 metric 100
10.1.0.0/24 dev UPLINK proto kernel scope link src 10.1.0.1
10.2.0.0/24 dev br1 proto kernel scope link src 10.2.0.3
10.3.0.0/24 dev enp7s0 proto kernel scope link src 10.3.0.3
10.98.30.0/24 via 10.2.0.200 dev br1
10.98.30.0/24 dev br0 proto kernel scope link src 10.98.30.148 metric 100
10.98.30.1 dev br0 proto dhcp scope link src 10.98.30.148 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
240.0.0.0/8 dev lxdfan0 proto kernel scope link src 240.148.0.1
v3
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
link/ether 00:16:3e:76:d8:df brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br1 state UP group default qlen 1000
link/ether 00:16:3e:ac:c2:6b brd ff:ff:ff:ff:ff:ff
4: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:16:3e:c1:bb:f7 brd ff:ff:ff:ff:ff:ff
inet 10.3.0.4/24 brd 10.3.0.255 scope global enp7s0
valid_lft forever preferred_lft forever
inet6 fd42:8fdd:7c0e:87c1:216:3eff:fec1:bbf7/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 3281sec preferred_lft 3281sec
inet6 fe80::216:3eff:fec1:bbf7/64 scope link
valid_lft forever preferred_lft forever
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 7a:f5:d5:3b:a3:46 brd ff:ff:ff:ff:ff:ff
inet 10.98.30.253/24 metric 100 brd 10.98.30.255 scope global dynamic br0
valid_lft 2704sec preferred_lft 2704sec
inet6 fd42:7386:9481:19b7:78f5:d5ff:fe3b:a346/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 3173sec preferred_lft 3173sec
inet6 fe80::78f5:d5ff:fe3b:a346/64 scope link
valid_lft forever preferred_lft forever
6: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether fa:70:2b:11:50:f8 brd ff:ff:ff:ff:ff:ff
inet 10.2.0.4/24 brd 10.2.0.255 scope global br1
valid_lft forever preferred_lft forever
inet6 fe80::f870:2bff:fe11:50f8/64 scope link
valid_lft forever preferred_lft forever
7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether ee:74:b7:c8:7a:46 brd ff:ff:ff:ff:ff:ff
9: br-int: <BROADCAST,MULTICAST> mtu 1442 qdisc noop state DOWN group default qlen 1000
link/ether 9e:11:82:e3:c4:e9 brd ff:ff:ff:ff:ff:ff
10: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
link/ether 86:bc:76:e1:84:a9 brd ff:ff:ff:ff:ff:ff
inet6 fe80::84ed:fbff:fe6c:8312/64 scope link
valid_lft forever preferred_lft forever
11: lxdovn2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether e6:71:42:28:fb:4e brd ff:ff:ff:ff:ff:ff
12: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:25:78:a9:ee brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
13: UPLINK: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:3f:93:c3 brd ff:ff:ff:ff:ff:ff
inet 10.1.0.1/24 scope global UPLINK
valid_lft forever preferred_lft forever
14: lxdfan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:37:1a:2e brd ff:ff:ff:ff:ff:ff
inet 240.253.0.1/8 scope global lxdfan0
valid_lft forever preferred_lft forever
19: lxdovn2b@lxdovn2a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
link/ether 26:0c:06:38:79:59 brd ff:ff:ff:ff:ff:ff
20: lxdovn2a@lxdovn2b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master UPLINK state UP group default qlen 1000
link/ether 8a:e9:65:9c:53:e6 brd ff:ff:ff:ff:ff:ff
21: lxdfan0-mtu: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
link/ether a2:9a:49:f8:75:28 brd ff:ff:ff:ff:ff:ff
inet6 fe80::a09a:49ff:fef8:7528/64 scope link
valid_lft forever preferred_lft forever
22: lxdfan0-fan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
link/ether e2:61:a0:eb:d5:af brd ff:ff:ff:ff:ff:ff
inet6 fe80::e061:a0ff:feeb:d5af/64 scope link
valid_lft forever preferred_lft forever
24: veth037a5ca0@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc noqueue master ovs-system state UP group default qlen 1000
link/ether 86:22:23:f0:66:58 brd ff:ff:ff:ff:ff:ff link-netnsid 0
25: lxdovn7b@lxdovn7a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
link/ether 92:6e:f3:c8:31:48 brd ff:ff:ff:ff:ff:ff
26: lxdovn7a@lxdovn7b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1 state UP group default qlen 1000
link/ether 56:1b:1c:f7:11:cb brd ff:ff:ff:ff:ff:ff
27: lxdovn7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether a2:51:88:4c:f8:48 brd ff:ff:ff:ff:ff:ff
default via 10.98.30.1 dev br0 proto dhcp src 10.98.30.253 metric 100
10.1.0.0/24 dev UPLINK proto kernel scope link src 10.1.0.1
10.2.0.0/24 dev br1 proto kernel scope link src 10.2.0.4
10.3.0.0/24 dev enp7s0 proto kernel scope link src 10.3.0.4
10.98.30.0/24 via 10.2.0.200 dev br1
10.98.30.0/24 dev br0 proto kernel scope link src 10.98.30.253 metric 100
10.98.30.1 dev br0 proto dhcp scope link src 10.98.30.253 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
240.0.0.0/8 dev lxdfan0 proto kernel scope link src 240.253.0.1
lxc network show dummy
config:
ipv4.gateway: 10.2.0.2/24
ipv4.ovn.ranges: 10.2.0.200-10.2.0.254
ipv4.routes: 10.98.30.0/24
volatile.last_state.created: "false"
description: ""
name: dummy
type: physical
used_by:
- /1.0/networks/ovn-virtual-network
managed: true
status: Created
locations:
- v3
- v1
- v2
lxc network show ovn-virtual-network
config:
bridge.mtu: "1442"
ipv4.address: 10.111.28.1/24
ipv4.nat: "true"
ipv6.address: fd42:8b63:b46:a6ea::1/64
ipv6.nat: "true"
network: dummy
volatile.network.ipv4.address: 10.2.0.200
description: ""
name: ovn-virtual-network
type: ovn
used_by:
- /1.0/instances/t1
managed: true
status: Created
locations:
- v1
- v2
- v3
lxc network forward list ovn-virtual-network
+----------------+-------------+------------------------+-------+----------+
| LISTEN ADDRESS | DESCRIPTION | DEFAULT TARGET ADDRESS | PORTS | LOCATION |
+----------------+-------------+------------------------+-------+----------+
| 10.98.30.30 | | 10.111.28.3 | 0 | |
+----------------+-------------+------------------------+-------+----------+
lxc network forward show ovn-virtual-network 10.98.30.30
description: ""
config:
target_address: 10.111.28.3
ports: []
listen_address: 10.98.30.30
location: ""
root@v1:~# ping 10.98.30.30
PING 10.98.30.30 (10.98.30.30) 56(84) bytes of data.
64 bytes from 10.98.30.30: icmp_seq=1 ttl=63 time=3.14 ms
^C
--- 10.98.30.30 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 3.143/3.143/3.143/0.000 ms
root@v2:~# ping 10.98.30.30
PING 10.98.30.30 (10.98.30.30) 56(84) bytes of data.
64 bytes from 10.98.30.30: icmp_seq=1 ttl=63 time=8.66 ms
^C
--- 10.98.30.30 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 8.664/8.664/8.664/0.000 ms
root@v3:~# ping 10.98.30.30
PING 10.98.30.30 (10.98.30.30) 56(84) bytes of data.
64 bytes from 10.98.30.30: icmp_seq=1 ttl=63 time=3.22 ms
^C
--- 10.98.30.30 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 3.217/3.217/3.217/0.000 ms
Failure bare metal machine hosting v1 v2 v3
ping 10.98.30.30
PING 10.98.30.30 (10.98.30.30) 56(84) bytes of data.
From 10.98.30.1 icmp_seq=1 Destination Host Unreachable
From 10.98.30.1 icmp_seq=3 Destination Host Unreachable
From 10.98.30.1 icmp_seq=4 Destination Host Unreachable
From 10.98.30.1 icmp_seq=5 Destination Host Unreachable
From 10.98.30.1 icmp_seq=6 Destination Host Unreachable
From 10.98.30.1 icmp_seq=7 Destination Host Unreachable
From 10.98.30.1 icmp_seq=8 Destination Host Unreachable
^C
--- 10.98.30.30 ping statistics ---
9 packets transmitted, 0 received, +7 errors, 100% packet loss, time 8135ms
pipe 4
ping 10.98.30.10
PING 10.98.30.10 (10.98.30.10) 56(84) bytes of data.
64 bytes from 10.98.30.10: icmp_seq=1 ttl=64 time=0.470 ms
64 bytes from 10.98.30.10: icmp_seq=2 ttl=64 time=0.431 ms
^C
--- 10.98.30.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1020ms
rtt min/avg/max/mdev = 0.431/0.450/0.470/0.019 ms
I am not seeing anything blocked in the firewall logs… but that doesnt mean I havent missed something.
hrmm another clue I missed earlier outbound pinging from the container in the v1/v2/v3 cluster isnt working either
root@v1:~# lxc shell t1
root@t1:~# ping google.com
^C
root@t1:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
I am guessing that the disconnect may be related to the br1 → dummy bridge setup thing I did… I dont have nat capability there etc…
The big gotcha is that I am trying to avoid using the main /28 ips that the isp will give me so that I can use them for network forwards or load balancers.
I think i was pretty close considering I can connect to everything from the v1/v2/v3 vms into the test container via the network forward.
but I just need to confirm that external devices can connect through v1/v2/v3 …
thanks
if there is a better/easier way to achieve this goal I am open to that suggestion.
my goals are to use clustering for maintenance/availability and preserve as many /28 ips on the external interface as possible. Hence my journey into ceph/ovn… instead of just running a single lxd instance.
Right, I’ve tried to figure out your setup, but the sheer amount of interfaces at every level is making it hard for me to see the woods from the trees. I don’t really understand why you have so many bridges and other interfaces setup.
But anyway, here’s what I think you’ve setup so far:
10.2.0.1/24.br1 with IPs 10.2.0.2/24, 10.2.0.3/24, 10.2.0.4/24 respectively.dummy that uses br1 as the parent (I can’t see that bit, I’m assuming, as would need to see the output of lxc network show dummy --target=<vm> as the parent interface is per-member).dummy network has 10.2.0.200-10.2.0.254 set for OVN routers, and 10.2.0.2/24 as the default gateway. It also has ipv4.routes: 10.98.30.0/24 set, but its not clear why thats needed.ovn-virtual-network that has an internal subnet of 10.111.28.0/24 and an external router address of 10.2.0.200.So in summary the network layout is as follows:
Physical host <-> lxdbr2 (10.2.0.1) <-> LXD VMs <-> br1 (10.2.0.2-4) <-> LXD OVN router (10.2.0.200) <-> LXD OVN network (10.111.28.1) <-> LXD instances (10.111.28.0/24).
So the first thing I can see that is wrong is that the dummy ipv4.gateway address is wrong.
Its set to the IP 10.2.0.2 of one of the VMs, rather than the IP of the physical host’s uplink bridge (10.2.0.1).
I also note you have docker installed both on the physical host and in the VMs.
While you are trying to diagnose this setup I would strongly recommend disabling docker (or removing it) so it doesn’t configure the respective host’s firewall, as its known to wreak havoc with LXD’s (or any other) bridge when forwarding traffic.
@tomp
This setup works…
`lxc network create UPLINK --type=physical parent=br0 --target v1
lxc network create UPLINK --type=physical parent=br0 --target v2
lxc network create UPLINK --type=physical parent=br0 --target v3
lxc network create UPLINK --type=physical
lxc network set UPLINK ipv4.routes=10.98.30.0/24
lxc network set UPLINK ipv4.gateway=10.98.30.1/24 dns.nameservers=8.8.8.8
lxc network set UPLINK ipv4.ovn.ranges=10.98.30.200-10.98.30.254
lxc network create my-ovn --type=ovn network=UPLINK
ipv4.address=10.1.1.1/24
ipv4.nat=true
dns.domain=foo.net
lxc network forward create my-ovn 10.98.30.2 target_address=“10.1.1.2”
lxc network load-balancer create my-ovn 10.98.30.3
lxc network load-balancer backend add my-ovn 10.98.30.3 dns1 10.1.1.2
lxc network load-balancer port add my-ovn 10.98.30.3 tcp 1234 dns1
lxc launch images:ubuntu/22.04 u1 --network my-ovn
`
however In my real production environment I will be limited on the uplink network to 14 total usable ips … and this means that I am using one 1 for the ovn router interface.
I was hopeing to do something like
br0 real uplink interface eth0 – connected to external gateway
br1 internal shared l2 between 3 vms eth1
`lxc network create UPLINK --type=physical parent=br1 --target v1
lxc network create UPLINK --type=physical parent=br1 --target v2
lxc network create UPLINK --type=physical parent=br1 --target v3
lxc network create UPLINK --type=physical
lxc network set UPLINK ipv4.routes=10.98.30.0/24
lxc network set UPLINK ipv4.gateway=10.98.30.1/24 dns.nameservers=8.8.8.8
lxc network set UPLINK ipv4.ovn.ranges=10.1.0.200-10.1.0.254
lxc network create my-ovn --type=ovn network=UPLINK
ipv4.address=10.1.1.1/24
ipv4.nat=true
dns.domain=foo.net
lxc network forward create my-ovn 10.98.30.2 target_address=“10.1.1.2”
lxc network load-balancer create my-ovn 10.98.30.3
lxc network load-balancer backend add my-ovn 10.98.30.3 dns1 10.1.1.2
lxc network load-balancer port add my-ovn 10.98.30.3 tcp 1234 dns1
lxc launch images:ubuntu/22.04 u1 --network my-ovn
`
but I am not able to get the second scenario to work. I am guessing I have to do some additional routing or something…
the second method has advantages in being able to create multiple ovn networks and not consume externally facing ips.
This second method is what I am trying to get documented/working.
Thanks so much for your help.
You can use the OVN router’s IP as one of the addresses for the network forwards too btw.
You can use lxdbr0 as the uplink for an OVN network on each LXD server, and assuming that NAT is enabled on lxdbr0, then outbound traffic will be SNATted to the external IP of the LXD server that is the active OVN chassis.
This will naturally prevent inbound connections to the OVN network (including the OVN router) from external sources, so your network forwards won’t work.
I’m a bit unclear how you expect this to work from a networking perspective (leaving OVN aside for the moment). On the one hand you want to allocate OVN networks without using external IPs (possible) but on the other hand you want to setup OVN network forwarders (possible too, but then using external IPs).
So I’m confused what you’re trying to achieve really 
I was trying to save as many uplink ips as possible for services. With your statement that forwards can use the ovn router IP I can accomplish that goal. Thanks for the advice.