OK so thats not really acceptable from an OVN HA perspective. In LXD each OVN network can potentially use any of the LXD cluster members as the gateway onto the uplink network. When the network is created each cluster member is assigned a random priority, and the highest active member becomes the active gateway chassis. If one cluster member fails the member with the next highest priority becomes the active gateway chassis etc.
So that means that OVN needs a shared L2 uplink network to operate in, as the external IP cannot be routed to only one cluster member (well it can, but if that turns out to be not the activate gateway chassis at some point in the future then connectivity will stop working).
What I suggest you do is leave OVN using the private local lxdbr0 bridge as its uplink, and then add firewall rules on each LXD cluster member to perform custom SNAT to the correct source IP for that machine based on the source address of the OVN network’s virtual router ( volatile.network.ipv{n}.address).
You’ll need to bind the external IPs to the specific LXD host’s external interface.
This setup will still be problematic as if you only have 2 IPs then what happens when the active chassis becomes a different server?