opened 08:30AM - 05 Apr 22 UTC
The [LXD](https://github.com/lxc/lxd) team use OVN load balancers for its Networ…k forward functionality (https://linuxcontainers.org/lxd/docs/master/howto/network_forwards/).
We have had a [user report](https://discuss.linuxcontainers.org/t/cant-make-network-forward-working-in-cluster-environment/13579) (since confirmed) from @lepokle that recent versions of OVN have broken the ARP responder functionality of OVN load balancers.
@lepokle has identified that it appears to broken since 21.06 specifically the commit https://github.com/ovn-org/ovn/commit/ea6ee901ff9107a084bc830a8a38c4e0bd9f75f7
We no longer see the OVN logical router responding to ARP requests for the load balancer listen IP on the chassis's gateway port.
If packets are manually routed to the OVN logical router's gateway port IP then the load balancer does still work though, so its just the ARP responder that has been broken.
On a pre-21.06 OVN version we see the following logical flows configured with a load balancer:
```
# ovn-nbctl --version
ovn-nbctl 21.03.0
Open vSwitch Library 2.15.90
DB Schema 5.31.0
# ovn-sbctl list logical_flow | grep bbb.76.20.84
match : "ct.est && ip && ip4.dst == bbb.76.20.84 && is_chassis_resident(\"cr-lxd-net11-lr-lrp-ext\")"
actions : "eth.dst = eth.src; eth.src = xreg0[0..47]; arp.op = 2; /* ARP reply */ arp.tha = arp.sha; arp.sha = xreg0[0..47]; arp.tpa = arp.spa; arp.spa = bbb.76.20.84; outport = inport; flags.loopback = 1; output;"
match : "inport == \"lxd-net11-lr-lrp-int\" && arp.op == 1 && arp.tpa == bbb.76.20.84"
match : "ip && ip4.dst == bbb.76.20.84"
actions : "reg1 = bbb.76.20.84; ct_lb(backends=10.161.64.2);"
match : "ct.new && ip4.dst == bbb.76.20.84"
match : "ct.new && ip && ip4.dst == bbb.76.20.84 && is_chassis_resident(\"cr-lxd-net11-lr-lrp-ext\")"
actions : "eth.dst = eth.src; eth.src = xreg0[0..47]; arp.op = 2; /* ARP reply */ arp.tha = arp.sha; arp.sha = xreg0[0..47]; arp.tpa = arp.spa; arp.spa = bbb.76.20.84; outport = inport; flags.loopback = 1; output;"
match : "inport == \"lxd-net11-lr-lrp-ext\" && arp.op == 1 && arp.tpa == bbb.76.20.84 && is_chassis_resident(\"cr-lxd-net11-lr-lrp-ext\")"
```
Specifically there is a mention of an `/* ARP reply */` flow.
However in later versions we see the ARP reply flow is gone:
```
# ovn-nbctl --version
ovn-nbctl 21.06.0
Open vSwitch Library 2.15.90
DB Schema 5.32.0
# ovn-sbctl list logical_flow | grep bbb.76.20.84
match : "ct.est && ip && ip4.dst == bbb.76.20.84 && is_chassis_resident(\"cr-lxd-net11-lr-lrp-ext\")"
match : "ip && ip4.dst == bbb.76.20.84"
match : "inport == \"lxd-net11-lr-lrp-int\" && arp.op == 1 && arp.tpa == { bbb.76.20.84 }"
actions : "reg1 = bbb.76.20.84; ct_lb(backends=10.161.64.2);"
match : "ct.new && ip4.dst == bbb.76.20.84"
match : "inport == \"lxd-net11-lr-lrp-ext\" && arp.op == 1 && arp.tpa == { bbb.76.20.84 } && is_chassis_resident(\"cr-lxd-net11-lr-lrp-ext\")"
match : "ct.new && ip && ip4.dst == bbb.76.20.84 && is_chassis_resident(\"cr-lxd-net11-lr-lrp-ext\")"
```