Think NVidia drivers are a bit tricky given how they are licensed and they require a lot of more diskspace which makes it tricky to install on a default IncusOS system.
One option would be to use a VM, pass through the GPU, install latest driver and Incus into it and run all GPU container in this VM. This solution would also separate issues where the driver could impact the host. It isn’t optimal but works.
Not sure if full NVidia support is planed, last time I checked Github it wasn’t.