Yes, it needs to be the same. As stgraber mentioned, the Incus-UI is just client-side files, it cannot proxy any backend calls. The main API port does all the works, including authentication.
To have better protection, I add Cloudflare Access to only allow me to reach the SSO login screen. Other people will be gated by Cloudflare - not even reach my server and see the Incus login screen.
Let me try to understand the concern here. Is it about external auth provider is aware of your internal API and doing some bad things?