SPICE with LXD is run over the LXD API, it is not available on a separate per-VM port or similar unsecure socket.
For setup like yours, what you should do is configure LXD to listen on the network (lxc config set core.https_address :8443), create a trust token (lxc config trust add --name your-client) and then on a desktop system, do lxc remote add your-server TOKEN to add the remote server to it.
With that done, you can do lxc remote switch your-server so all operations from that remote client are run against your server, run lxc list to confirm it’s all good and then you can run lxc console --type vga your-vm and you’ll get remote-viewer on your desktop system showing you the remote VM on your server.