Can you show output of lxc remote list and also please can you enable the debug logging on the target LXD server and show the output of the logs here when you try to add it as a s remote:
That does not show anything. I figured out why lxd was not reachable. Haproxy was using port 8443. The strange thing is that lxd does not complain about this port.