A working theory I have at the moment is that LXD is trying to connect on all of the available IPs sequentially (which it does in some places, although I’m not certain this is one of them, although its a strong possibility given what we are seeing), and this can take time as some IPs are not listening (perhaps a firewall is blocking the request causing a timeout rather than connection refused).
Previously it would have kept trying until it found the right one, but with the listener timeouts added in 4.20, the timeout is hit before the correct IP is found, and by the time the destination member does try and connect on it the websocket on the source has been closed.