Sovled
This was an issue with the host pcscd process claiming the smartcard and preventing the container from claiming it. I disabled pcscd on the host and the container worked -
systemctl stop pcscd.socket
systemctl stop pcscd
Looks like unless pcscd is specifically needed, it may not even need to be installed and scdaemon can access the card directly.