One thing you could do to try and prove this theory is to make sure that inbound requests to that IPv6 address are rejected rather than dropped (if they are now) so that LXD destination will immediately move onto the next IP. If that works it’ll prove my theory.