If you copied the client.crt from the Incus server and added it to the trust list of the LXD server, then the Incus client should now be able to add your LXD server (not the reverse).
Tokens will not work, so you’ll need to just use incus remote add lxd-server https://IP:8443 it should prompt to confirm the server fingerprint and then get added successfully as the server will already be trusting the certificate.
If you want the reverse to work, then you need to transfer the client.crt from your user on the LXD server (look for it somewhere under ~/snap/lxd/common), then transfer it to the Incus server and get it trusted there. After that lxc remote add incus-server https://IP:8443 should similarly work.