Is there a video for lxd clustering with ovn and forwards?

Is there an advanced ingress video which covers ovn?

I see vids covering network bridges and a single node.

I am looking for something that coves ovn. Discusses setting a shared bridge on the uplink nic or how your supposed to do that. Covers trouble shooting and where to look for issues… eg firewall logging. Or ovn logs. What is the traffic flow for ingress etc?

What might cause the proxy to only work one etc. Does the controller change between chassis etc

We have https://www.youtube.com/watch?v=B-Uzo9WldMs but I expect you’ve seen that already.

We have https://www.youtube.com/watch?v=1M__Rm9iZb8 for general OVN cluster setup (including uplink).

What is the issue you are encountering?

I am struggling with consistent ingress for forwards and proxies. The proxy works once and the forward not at all.

I have setup a standalone bridge on each instance.
I set the external subnet as a route on this dummy bridge.

My test vms have 3 physiacal nics. One for internal ovn traffic. One for ceph traffic and one that is the external interface.

I created a new bridge called uplink. And a new ovn network that uses the uplink bridge as its uplink and uses nat for outbound traffic.

I am trying to simulate the hardware i will request from the isp to make this work.

External nic. /28 14 usable ips but 3 will be used for lxd nodes.
Ovn nic (internal). 10.40.0.x
Ceph nic (internal). 10.41.0.x
Uplink (br) (lxd managed)(no hardware) 10.0.0.x. → lxdbr0(lxd managed)(ovn). 10.1.1.x

Do you have a LXD cluster for OVN or a single LXD server?

Its a cluster.
I setup 3 vms as a cluster.

OK so LXD’s OVN networks require a shared L2 for the uplink network.

See https://linuxcontainers.org/lxd/docs/master/howto/network_ovn_setup/

A high availability OVN cluster requires a shared layer 2 network, so that the active OVN chassis can move between cluster members (which effectively allows the OVN router’s external IP to be reachable from a different host).
Therefore, you must specify either an unmanaged bridge interface or an unused physical interface as the parent for the physical network that is used for OVN uplink.

Using a LXD managed private bridge for the OVN uplink is OK for standalone servers and testing, but no good for clustered setups.

This is because OVN will select a single cluster member to be used for the uplink connection for instances on all the LXD cluster members.

Are you suggesting another internal nic for this traffic?

Yes, or an unmanaged bridge that is connected to the same external network as each of the cluster members.

@tomp I think I am very close now. from my v1 v2 v3 instances I can now connect to a port and see data transfer… however the last piece is not working I cannot see the assigned ip from the host that is hosting v1 v2 v3 … I can see the bridged interfaces

the host bridge does own the entire subnet…

Please show ip a and ip r output from v1, v2 and v3, along with the host you’re testing from.
Also please show lxc network show <uplink> and lxc network show <ovn>

Underlying host physical hardware
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp7s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:ae:c5:62:dd:54 brd ff:ff:ff:ff:ff:ff
    inet 192.168.5.6/24 brd 192.168.5.255 scope global noprefixroute enp7s1
       valid_lft forever preferred_lft forever
    inet6 2001:470:4470::18e/128 scope global dynamic noprefixroute 
       valid_lft 39131sec preferred_lft 39131sec
    inet6 fd23:c1f1:f9c::18e/128 scope global dynamic noprefixroute 
       valid_lft 39131sec preferred_lft 39131sec
    inet6 fd23:c1f1:f9c:0:4694:61e3:c642:5fe/64 scope global temporary dynamic 
       valid_lft 540297sec preferred_lft 21307sec
    inet6 fd23:c1f1:f9c:0:1cfe:7600:ec7d:c078/64 scope global mngtmpaddr noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 2001:470:4470:0:91fc:127f:e2b7:876d/64 scope global temporary dynamic 
       valid_lft 540297sec preferred_lft 21307sec
    inet6 2001:470:4470:0:2e80:bbf2:ea15:6c0c/64 scope global mngtmpaddr noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::75f5:34c3:8e9c:49f7/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:e8:18:0b brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: lxcbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.1/24 brd 10.0.3.255 scope global lxcbr0
       valid_lft forever preferred_lft forever
5: zthnhhq2ry: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2800 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether 0e:99:ef:d6:76:74 brd ff:ff:ff:ff:ff:ff
    inet 192.168.191.183/24 brd 192.168.191.255 scope global zthnhhq2ry
       valid_lft forever preferred_lft forever
    inet 192.168.191.186/24 brd 192.168.191.255 scope global secondary zthnhhq2ry
       valid_lft forever preferred_lft forever
    inet6 fdaf:78bf:9436:c7fe:e99:9367:28e0:e2cb/88 scope global 
       valid_lft forever preferred_lft forever
    inet6 fc99:bf41:9a67:28e0:e2cb::1/40 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::c99:efff:fed6:7674/64 scope link 
       valid_lft forever preferred_lft forever
6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:6f:83:e9:82 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
7: br-0c9840d94d0e: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:3d:51:86:a3 brd ff:ff:ff:ff:ff:ff
    inet 172.23.0.1/16 brd 172.23.255.255 scope global br-0c9840d94d0e
       valid_lft forever preferred_lft forever
8: br-c45814d16427: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:4d:4e:00:22 brd ff:ff:ff:ff:ff:ff
    inet 172.27.0.1/16 brd 172.27.255.255 scope global br-c45814d16427
       valid_lft forever preferred_lft forever
9: br-c8bd14a80735: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:ce:5b:12:2e brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-c8bd14a80735
       valid_lft forever preferred_lft forever
10: mpqemubr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:35:84:28 brd ff:ff:ff:ff:ff:ff
    inet 10.175.149.1/24 brd 10.175.149.255 scope global mpqemubr0
       valid_lft forever preferred_lft forever
11: rh-priv: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:1c:5c:25 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/24 scope global rh-priv
       valid_lft forever preferred_lft forever
12: rh-pub: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:8c:5e:56 brd ff:ff:ff:ff:ff:ff
    inet 10.196.125.1/28 scope global rh-pub
       valid_lft forever preferred_lft forever
13: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:e5:7c:c5 brd ff:ff:ff:ff:ff:ff
    inet 10.96.124.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:2c94:34b6:cabc::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fee5:7cc5/64 scope link 
       valid_lft forever preferred_lft forever
14: lxdbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:e8:41:17 brd ff:ff:ff:ff:ff:ff
    inet 10.98.30.1/24 scope global lxdbr1
       valid_lft forever preferred_lft forever
    inet6 fd42:7386:9481:19b7::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fee8:4117/64 scope link 
       valid_lft forever preferred_lft forever
15: lxdbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:f6:cf:35 brd ff:ff:ff:ff:ff:ff
    inet 10.2.0.1/24 scope global lxdbr2
       valid_lft forever preferred_lft forever
16: lxdbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:32:26:bf brd ff:ff:ff:ff:ff:ff
    inet 10.3.0.1/24 scope global lxdbr3
       valid_lft forever preferred_lft forever
    inet6 fd42:8fdd:7c0e:87c1::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe32:26bf/64 scope link 
       valid_lft forever preferred_lft forever
18: veth2edd9d8e@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 8a:34:65:16:19:e8 brd ff:ff:ff:ff:ff:ff link-netnsid 0
19: tap1f26c73e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr0 state UP group default qlen 1000
    link/ether 7e:b1:e1:eb:8b:f7 brd ff:ff:ff:ff:ff:ff
21: vethde5dd531@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 8a:0f:ff:68:e6:8f brd ff:ff:ff:ff:ff:ff link-netnsid 1
22: tap00a8bc43: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr1 state UP group default qlen 1000
    link/ether 6e:d4:b7:ad:7d:30 brd ff:ff:ff:ff:ff:ff
23: tap081b9deb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr2 state UP group default qlen 1000
    link/ether 8a:b6:9c:ef:a4:4d brd ff:ff:ff:ff:ff:ff
24: tap8fdd3773: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr3 state UP group default qlen 1000
    link/ether 86:90:f3:5e:e1:16 brd ff:ff:ff:ff:ff:ff
25: tap59dff295: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr1 state UP group default qlen 1000
    link/ether 6a:ef:f7:a3:eb:4f brd ff:ff:ff:ff:ff:ff
26: tap7224ac40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr2 state UP group default qlen 1000
    link/ether ba:99:0f:c5:de:45 brd ff:ff:ff:ff:ff:ff
27: tap11c82ce6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr3 state UP group default qlen 1000
    link/ether 2e:df:69:1d:82:10 brd ff:ff:ff:ff:ff:ff
28: tap3759d2a8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr1 state UP group default qlen 1000
    link/ether c2:3b:cc:b1:89:36 brd ff:ff:ff:ff:ff:ff
29: tapc2d7bc89: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr2 state UP group default qlen 1000
    link/ether 96:76:9f:98:11:4e brd ff:ff:ff:ff:ff:ff
30: tap350478e9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lxdbr3 state UP group default qlen 1000
    link/ether 4a:23:53:d8:42:e5 brd ff:ff:ff:ff:ff:ff
32: veth7511099d@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether c2:3d:2b:62:f4:46 brd ff:ff:ff:ff:ff:ff link-netnsid 2


 ip r
default via 192.168.5.1 dev enp7s1 proto static metric 100 
10.0.3.0/24 dev lxcbr0 proto kernel scope link src 10.0.3.1 linkdown 
10.1.0.0/24 dev rh-priv proto kernel scope link src 10.1.0.1 linkdown 
10.2.0.0/24 dev lxdbr2 proto kernel scope link src 10.2.0.1 
10.3.0.0/24 dev lxdbr3 proto kernel scope link src 10.3.0.1 
10.96.124.0/24 dev lxdbr0 proto kernel scope link src 10.96.124.1 
10.98.30.0/24 dev lxdbr1 proto kernel scope link src 10.98.30.1 
10.175.149.0/24 dev mpqemubr0 proto kernel scope link src 10.175.149.1 linkdown 
10.196.125.0/28 dev rh-pub proto kernel scope link src 10.196.125.1 linkdown 
169.254.0.0/16 dev virbr0 scope link metric 1000 linkdown 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.18.0.0/16 dev br-c8bd14a80735 proto kernel scope link src 172.18.0.1 linkdown 
172.23.0.0/16 dev br-0c9840d94d0e proto kernel scope link src 172.23.0.1 linkdown 
172.27.0.0/16 dev br-c45814d16427 proto kernel scope link src 172.27.0.1 linkdown 
192.168.5.0/24 dev enp7s1 proto kernel scope link src 192.168.5.6 metric 100 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown 
192.168.191.0/24 dev zthnhhq2ry proto kernel scope link src 192.168.191.183 
for vm in v1 v2 v3; do echo $vm;echo ;lxc exec $vm ip a;echo; lxc exec $vm ip r; done
v1

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether 00:16:3e:58:b9:82 brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br1 state UP group default qlen 1000
    link/ether 00:16:3e:a0:29:b0 brd ff:ff:ff:ff:ff:ff
4: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:96:bd:bf brd ff:ff:ff:ff:ff:ff
    inet 10.3.0.2/24 brd 10.3.0.255 scope global enp7s0
       valid_lft forever preferred_lft forever
    inet6 fd42:8fdd:7c0e:87c1:216:3eff:fe96:bdbf/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 3282sec preferred_lft 3282sec
    inet6 fe80::216:3eff:fe96:bdbf/64 scope link 
       valid_lft forever preferred_lft forever
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f6:da:31:98:57:7c brd ff:ff:ff:ff:ff:ff
    inet 10.98.30.10/24 metric 100 brd 10.98.30.255 scope global dynamic br0
       valid_lft 2806sec preferred_lft 2806sec
    inet6 fd42:7386:9481:19b7:f4da:31ff:fe98:577c/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 3174sec preferred_lft 3174sec
    inet6 fe80::f4da:31ff:fe98:577c/64 scope link 
       valid_lft forever preferred_lft forever
6: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 62:4c:bb:3c:cd:b7 brd ff:ff:ff:ff:ff:ff
    inet 10.2.0.2/24 brd 10.2.0.255 scope global br1
       valid_lft forever preferred_lft forever
    inet6 fe80::604c:bbff:fe3c:cdb7/64 scope link 
       valid_lft forever preferred_lft forever
7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 36:ac:58:2e:66:4e brd ff:ff:ff:ff:ff:ff
9: br-int: <BROADCAST,MULTICAST> mtu 1442 qdisc noop state DOWN group default qlen 1000
    link/ether 0a:e4:16:7a:61:1f brd ff:ff:ff:ff:ff:ff
10: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether 86:2f:f3:46:3a:0a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::10cb:e6ff:fe00:2e10/64 scope link 
       valid_lft forever preferred_lft forever
11: lxdovn2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 06:e5:a6:f1:9f:45 brd ff:ff:ff:ff:ff:ff
12: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:3b:f4:46:f4 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
13: lxdfan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:fd:4c:cf brd ff:ff:ff:ff:ff:ff
    inet 240.10.0.1/8 scope global lxdfan0
       valid_lft forever preferred_lft forever
16: UPLINK: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:3f:93:c3 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/24 scope global UPLINK
       valid_lft forever preferred_lft forever
17: lxdovn2b@lxdovn2a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 6e:fc:20:37:36:f7 brd ff:ff:ff:ff:ff:ff
18: lxdovn2a@lxdovn2b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master UPLINK state UP group default qlen 1000
    link/ether ae:02:46:87:8d:42 brd ff:ff:ff:ff:ff:ff
21: lxdfan0-mtu: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
    link/ether 1a:8a:ff:2d:6b:f5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::703f:b3ff:fe70:951/64 scope link 
       valid_lft forever preferred_lft forever
22: lxdfan0-fan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
    link/ether 5a:66:26:d3:66:c4 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5866:26ff:fed3:66c4/64 scope link 
       valid_lft forever preferred_lft forever
24: veth8324f1a8@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether be:e7:8e:aa:f4:d7 brd ff:ff:ff:ff:ff:ff link-netnsid 0
25: lxdovn7b@lxdovn7a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether c2:73:2a:49:a3:43 brd ff:ff:ff:ff:ff:ff
26: lxdovn7a@lxdovn7b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1 state UP group default qlen 1000
    link/ether 7e:8b:06:d2:ac:ff brd ff:ff:ff:ff:ff:ff
27: lxdovn7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 12:84:c9:e9:22:48 brd ff:ff:ff:ff:ff:ff

default via 10.98.30.1 dev br0 proto dhcp src 10.98.30.10 metric 100 
10.1.0.0/24 dev UPLINK proto kernel scope link src 10.1.0.1 
10.2.0.0/24 dev br1 proto kernel scope link src 10.2.0.2 
10.3.0.0/24 dev enp7s0 proto kernel scope link src 10.3.0.2 
10.98.30.0/24 via 10.2.0.200 dev br1 
10.98.30.0/24 dev br0 proto kernel scope link src 10.98.30.10 metric 100 
10.98.30.1 dev br0 proto dhcp scope link src 10.98.30.10 metric 100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
240.0.0.0/8 dev lxdfan0 proto kernel scope link src 240.10.0.1 
v2

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether 00:16:3e:02:a8:7f brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br1 state UP group default qlen 1000
    link/ether 00:16:3e:cf:c8:e1 brd ff:ff:ff:ff:ff:ff
4: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:2f:5b:6a brd ff:ff:ff:ff:ff:ff
    inet 10.3.0.3/24 brd 10.3.0.255 scope global enp7s0
       valid_lft forever preferred_lft forever
    inet6 fd42:8fdd:7c0e:87c1:216:3eff:fe2f:5b6a/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 3281sec preferred_lft 3281sec
    inet6 fe80::216:3eff:fe2f:5b6a/64 scope link 
       valid_lft forever preferred_lft forever
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0e:d9:5f:19:7d:cd brd ff:ff:ff:ff:ff:ff
    inet 10.98.30.148/24 metric 100 brd 10.98.30.255 scope global dynamic br0
       valid_lft 3113sec preferred_lft 3113sec
    inet6 fd42:7386:9481:19b7:cd9:5fff:fe19:7dcd/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 3174sec preferred_lft 3174sec
    inet6 fe80::cd9:5fff:fe19:7dcd/64 scope link 
       valid_lft forever preferred_lft forever
6: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7e:f9:a7:8a:9f:29 brd ff:ff:ff:ff:ff:ff
    inet 10.2.0.3/24 brd 10.2.0.255 scope global br1
       valid_lft forever preferred_lft forever
    inet6 fe80::7cf9:a7ff:fe8a:9f29/64 scope link 
       valid_lft forever preferred_lft forever
7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d2:47:7d:e1:19:eb brd ff:ff:ff:ff:ff:ff
9: br-int: <BROADCAST,MULTICAST> mtu 1442 qdisc noop state DOWN group default qlen 1000
    link/ether 52:9c:3c:01:08:14 brd ff:ff:ff:ff:ff:ff
10: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether 1e:cc:e7:79:e1:15 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7c74:8aff:fe68:337d/64 scope link 
       valid_lft forever preferred_lft forever
11: lxdovn2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 92:3f:4d:b8:19:4a brd ff:ff:ff:ff:ff:ff
12: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:a3:3b:ce:eb brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
13: UPLINK: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:3f:93:c3 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/24 scope global UPLINK
       valid_lft forever preferred_lft forever
14: lxdfan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:4c:47:d5 brd ff:ff:ff:ff:ff:ff
    inet 240.148.0.1/8 scope global lxdfan0
       valid_lft forever preferred_lft forever
17: lxdovn2b@lxdovn2a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 5a:47:91:e1:19:32 brd ff:ff:ff:ff:ff:ff
18: lxdovn2a@lxdovn2b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master UPLINK state UP group default qlen 1000
    link/ether 26:4c:b4:b9:e4:0d brd ff:ff:ff:ff:ff:ff
21: lxdfan0-mtu: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
    link/ether 02:14:d6:71:58:3b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::14:d6ff:fe71:583b/64 scope link 
       valid_lft forever preferred_lft forever
22: lxdfan0-fan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
    link/ether 2e:9a:42:dd:84:ed brd ff:ff:ff:ff:ff:ff
    inet6 fe80::2c9a:42ff:fedd:84ed/64 scope link 
       valid_lft forever preferred_lft forever
24: veth079edabf@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 72:10:a2:df:ea:c0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
27: lxdovn7b@lxdovn7a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 62:2f:51:76:3f:4b brd ff:ff:ff:ff:ff:ff
28: lxdovn7a@lxdovn7b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1 state UP group default qlen 1000
    link/ether 5e:64:c0:8d:f0:31 brd ff:ff:ff:ff:ff:ff
29: lxdovn7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 3e:4b:6b:be:3e:4c brd ff:ff:ff:ff:ff:ff
33: veth4b08de91@if32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 92:3a:43:5a:6c:c6 brd ff:ff:ff:ff:ff:ff link-netnsid 2

default via 10.98.30.1 dev br0 proto dhcp src 10.98.30.148 metric 100 
10.1.0.0/24 dev UPLINK proto kernel scope link src 10.1.0.1 
10.2.0.0/24 dev br1 proto kernel scope link src 10.2.0.3 
10.3.0.0/24 dev enp7s0 proto kernel scope link src 10.3.0.3 
10.98.30.0/24 via 10.2.0.200 dev br1 
10.98.30.0/24 dev br0 proto kernel scope link src 10.98.30.148 metric 100 
10.98.30.1 dev br0 proto dhcp scope link src 10.98.30.148 metric 100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
240.0.0.0/8 dev lxdfan0 proto kernel scope link src 240.148.0.1 
v3

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether 00:16:3e:76:d8:df brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br1 state UP group default qlen 1000
    link/ether 00:16:3e:ac:c2:6b brd ff:ff:ff:ff:ff:ff
4: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:c1:bb:f7 brd ff:ff:ff:ff:ff:ff
    inet 10.3.0.4/24 brd 10.3.0.255 scope global enp7s0
       valid_lft forever preferred_lft forever
    inet6 fd42:8fdd:7c0e:87c1:216:3eff:fec1:bbf7/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 3281sec preferred_lft 3281sec
    inet6 fe80::216:3eff:fec1:bbf7/64 scope link 
       valid_lft forever preferred_lft forever
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:f5:d5:3b:a3:46 brd ff:ff:ff:ff:ff:ff
    inet 10.98.30.253/24 metric 100 brd 10.98.30.255 scope global dynamic br0
       valid_lft 2704sec preferred_lft 2704sec
    inet6 fd42:7386:9481:19b7:78f5:d5ff:fe3b:a346/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 3173sec preferred_lft 3173sec
    inet6 fe80::78f5:d5ff:fe3b:a346/64 scope link 
       valid_lft forever preferred_lft forever
6: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fa:70:2b:11:50:f8 brd ff:ff:ff:ff:ff:ff
    inet 10.2.0.4/24 brd 10.2.0.255 scope global br1
       valid_lft forever preferred_lft forever
    inet6 fe80::f870:2bff:fe11:50f8/64 scope link 
       valid_lft forever preferred_lft forever
7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ee:74:b7:c8:7a:46 brd ff:ff:ff:ff:ff:ff
9: br-int: <BROADCAST,MULTICAST> mtu 1442 qdisc noop state DOWN group default qlen 1000
    link/ether 9e:11:82:e3:c4:e9 brd ff:ff:ff:ff:ff:ff
10: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether 86:bc:76:e1:84:a9 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::84ed:fbff:fe6c:8312/64 scope link 
       valid_lft forever preferred_lft forever
11: lxdovn2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e6:71:42:28:fb:4e brd ff:ff:ff:ff:ff:ff
12: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:25:78:a9:ee brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
13: UPLINK: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:3f:93:c3 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/24 scope global UPLINK
       valid_lft forever preferred_lft forever
14: lxdfan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:37:1a:2e brd ff:ff:ff:ff:ff:ff
    inet 240.253.0.1/8 scope global lxdfan0
       valid_lft forever preferred_lft forever
19: lxdovn2b@lxdovn2a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 26:0c:06:38:79:59 brd ff:ff:ff:ff:ff:ff
20: lxdovn2a@lxdovn2b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master UPLINK state UP group default qlen 1000
    link/ether 8a:e9:65:9c:53:e6 brd ff:ff:ff:ff:ff:ff
21: lxdfan0-mtu: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
    link/ether a2:9a:49:f8:75:28 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a09a:49ff:fef8:7528/64 scope link 
       valid_lft forever preferred_lft forever
22: lxdfan0-fan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
    link/ether e2:61:a0:eb:d5:af brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e061:a0ff:feeb:d5af/64 scope link 
       valid_lft forever preferred_lft forever
24: veth037a5ca0@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 86:22:23:f0:66:58 brd ff:ff:ff:ff:ff:ff link-netnsid 0
25: lxdovn7b@lxdovn7a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 92:6e:f3:c8:31:48 brd ff:ff:ff:ff:ff:ff
26: lxdovn7a@lxdovn7b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1 state UP group default qlen 1000
    link/ether 56:1b:1c:f7:11:cb brd ff:ff:ff:ff:ff:ff
27: lxdovn7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a2:51:88:4c:f8:48 brd ff:ff:ff:ff:ff:ff

default via 10.98.30.1 dev br0 proto dhcp src 10.98.30.253 metric 100 
10.1.0.0/24 dev UPLINK proto kernel scope link src 10.1.0.1 
10.2.0.0/24 dev br1 proto kernel scope link src 10.2.0.4 
10.3.0.0/24 dev enp7s0 proto kernel scope link src 10.3.0.4 
10.98.30.0/24 via 10.2.0.200 dev br1 
10.98.30.0/24 dev br0 proto kernel scope link src 10.98.30.253 metric 100 
10.98.30.1 dev br0 proto dhcp scope link src 10.98.30.253 metric 100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
240.0.0.0/8 dev lxdfan0 proto kernel scope link src 240.253.0.1 
lxc network show dummy
config:
  ipv4.gateway: 10.2.0.2/24
  ipv4.ovn.ranges: 10.2.0.200-10.2.0.254
  ipv4.routes: 10.98.30.0/24
  volatile.last_state.created: "false"
description: ""
name: dummy
type: physical
used_by:
- /1.0/networks/ovn-virtual-network
managed: true
status: Created
locations:
- v3
- v1
- v2
lxc network show ovn-virtual-network
config:
  bridge.mtu: "1442"
  ipv4.address: 10.111.28.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:8b63:b46:a6ea::1/64
  ipv6.nat: "true"
  network: dummy
  volatile.network.ipv4.address: 10.2.0.200
description: ""
name: ovn-virtual-network
type: ovn
used_by:
- /1.0/instances/t1
managed: true
status: Created
locations:
- v1
- v2
- v3
lxc network forward list ovn-virtual-network
+----------------+-------------+------------------------+-------+----------+
| LISTEN ADDRESS | DESCRIPTION | DEFAULT TARGET ADDRESS | PORTS | LOCATION |
+----------------+-------------+------------------------+-------+----------+
| 10.98.30.30    |             | 10.111.28.3            | 0     |          |
+----------------+-------------+------------------------+-------+----------+

lxc network forward show ovn-virtual-network 10.98.30.30
description: ""
config:
  target_address: 10.111.28.3
ports: []
listen_address: 10.98.30.30
location: ""
root@v1:~# ping 10.98.30.30
PING 10.98.30.30 (10.98.30.30) 56(84) bytes of data.
64 bytes from 10.98.30.30: icmp_seq=1 ttl=63 time=3.14 ms
^C
--- 10.98.30.30 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 3.143/3.143/3.143/0.000 ms

root@v2:~# ping 10.98.30.30
PING 10.98.30.30 (10.98.30.30) 56(84) bytes of data.
64 bytes from 10.98.30.30: icmp_seq=1 ttl=63 time=8.66 ms
^C
--- 10.98.30.30 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 8.664/8.664/8.664/0.000 ms


root@v3:~# ping 10.98.30.30
PING 10.98.30.30 (10.98.30.30) 56(84) bytes of data.
64 bytes from 10.98.30.30: icmp_seq=1 ttl=63 time=3.22 ms
^C
--- 10.98.30.30 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 3.217/3.217/3.217/0.000 ms



Failure bare metal machine hosting v1 v2 v3
ping 10.98.30.30
PING 10.98.30.30 (10.98.30.30) 56(84) bytes of data.
From 10.98.30.1 icmp_seq=1 Destination Host Unreachable
From 10.98.30.1 icmp_seq=3 Destination Host Unreachable
From 10.98.30.1 icmp_seq=4 Destination Host Unreachable
From 10.98.30.1 icmp_seq=5 Destination Host Unreachable
From 10.98.30.1 icmp_seq=6 Destination Host Unreachable
From 10.98.30.1 icmp_seq=7 Destination Host Unreachable
From 10.98.30.1 icmp_seq=8 Destination Host Unreachable
^C
--- 10.98.30.30 ping statistics ---
9 packets transmitted, 0 received, +7 errors, 100% packet loss, time 8135ms
pipe 4
ping 10.98.30.10
PING 10.98.30.10 (10.98.30.10) 56(84) bytes of data.
64 bytes from 10.98.30.10: icmp_seq=1 ttl=64 time=0.470 ms
64 bytes from 10.98.30.10: icmp_seq=2 ttl=64 time=0.431 ms
^C
--- 10.98.30.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1020ms
rtt min/avg/max/mdev = 0.431/0.450/0.470/0.019 ms

I am not seeing anything blocked in the firewall logs… but that doesnt mean I havent missed something.

hrmm another clue I missed earlier outbound pinging from the container in the v1/v2/v3 cluster isnt working either

root@v1:~# lxc shell t1
root@t1:~# ping google.com
^C
root@t1:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C

I am guessing that the disconnect may be related to the br1 → dummy bridge setup thing I did… I dont have nat capability there etc…

The big gotcha is that I am trying to avoid using the main /28 ips that the isp will give me so that I can use them for network forwards or load balancers.

I think i was pretty close considering I can connect to everything from the v1/v2/v3 vms into the test container via the network forward.

but I just need to confirm that external devices can connect through v1/v2/v3 …
thanks

if there is a better/easier way to achieve this goal I am open to that suggestion.

my goals are to use clustering for maintenance/availability and preserve as many /28 ips on the external interface as possible. Hence my journey into ceph/ovn… instead of just running a single lxd instance.

Right, I’ve tried to figure out your setup, but the sheer amount of interfaces at every level is making it hard for me to see the woods from the trees. I don’t really understand why you have so many bridges and other interfaces setup.

But anyway, here’s what I think you’ve setup so far:

  • 1x physical host with a managed LXD bridge called lxdbr2 with an IP of 10.2.0.1/24.
  • 3x LXD VMs, connected to the host’s lxdbr2 bridge, and then inside the guest an unmanaged bridge called br1 with IPs 10.2.0.2/24, 10.2.0.3/24, 10.2.0.4/24 respectively.
  • The VMs are running a LXD cluster and have an physical network defined called dummy that uses br1 as the parent (I can’t see that bit, I’m assuming, as would need to see the output of lxc network show dummy --target=<vm> as the parent interface is per-member).
  • The dummy network has 10.2.0.200-10.2.0.254 set for OVN routers, and 10.2.0.2/24 as the default gateway. It also has ipv4.routes: 10.98.30.0/24 set, but its not clear why thats needed.
  • An OVN network called ovn-virtual-network that has an internal subnet of 10.111.28.0/24 and an external router address of 10.2.0.200.

So in summary the network layout is as follows:

Physical host <-> lxdbr2 (10.2.0.1) <-> LXD VMs <-> br1 (10.2.0.2-4) <-> LXD OVN router (10.2.0.200) <-> LXD OVN network (10.111.28.1) <-> LXD instances (10.111.28.0/24).

So the first thing I can see that is wrong is that the dummy ipv4.gateway address is wrong.
Its set to the IP 10.2.0.2 of one of the VMs, rather than the IP of the physical host’s uplink bridge (10.2.0.1).

I also note you have docker installed both on the physical host and in the VMs.
While you are trying to diagnose this setup I would strongly recommend disabling docker (or removing it) so it doesn’t configure the respective host’s firewall, as its known to wreak havoc with LXD’s (or any other) bridge when forwarding traffic.

See https://linuxcontainers.org/lxd/docs/master/howto/network_bridge_firewalld/#prevent-issues-with-lxd-and-docker