The error was extremely stupid on my side, it turned out that blacklisting nvidia drivers was not needed at all
The problem was a typo in vfio-pci.ids in my GRUB default cfg.
If anyone has the same problem I found a great article describing 4 ways to do that