Also, is VirGL worth checking out for this kind of use case?
Incus out of the box uses virtio-gpu-gl-pci (or Virtio-GPU GL or VirGL).
As to the rest of your post, what kind of environment are you looking for? If any of your devs require Windows then that narrows your options down significantly to only GPU passthrough. If they are fine with Linux then you have the gamut of Virtio-GPU GL or Venus (GL and Vulkan respectively). I’m not going to repeat myself so you can check the thread here for more details: Incus DE experience
EDIT: If you want Virtio-GPU Venus then you’ll need to be running a host that ships the latest kernels (6.14 or later) and has Qemu 9.2 and newer, and Virglrenderer 1.0.0 and newer
Then comes the question, if you are using Linux instances, of whether you want Wayland support or not. IIRC Gnome has the furthest support for headless Mutter and KDE is some degree behind, but both are suboptimal environments; so this would rule out system container instances. You may want to experiment on this yourself however as my knowledge is some months out of date.
If you do want Wayland then you’re pigeonholed into virtual machines.
As to whether Moonlight works with Virtio-GPU is something you’ll have to explore as I haven’t used it on non-PCIe passthrough’d GPUs. Otherwise you have SPICE and RDP (as in Gnome’s RDP and KDE’s krdp offerings)
Ideally, I’d like to build something on top of consumer-grade hardware or Hetzner-cost-style dedicated servers.
You will very quickly run out of PCIe lanes on consumer platforms unless you do some form of clustering with one GPU and one NIC (for optimal remote desktop and networked storage performance) per machine; you’d want Threadripper, Epyc, or one of the Xeon platforms.
Maybe the state of the art has changed and if so someone else can chime in.