I use this server as a htpc too, so I need a compositor on host. But you don’t have to.
If you just want a headless GUI CT, you don’t even need to read this post, just “incus launch images:debian/12 headlessguict”, “incus exec headlessguict bash”, “apt update && apt upgrade”, “apt install whateverDEyouwant”, “apt install whateverRDPorVNCyouwant”, create a user. Then as the new user config autostart vnc or rdp as the DE required. You can test now using cli, start the DE, after DE started, vnc or rdp also started, connect it, if succeeded read next, else fix it.
Now you will encounter a problem, incus doesn’t have a init system like docker, so after this CT start, it will do nothing, means your DE won’t start, your vnc or rdp won’t start, basically useless. So read this: https://discuss.linuxcontainers.org/t/autostart-container-desktop-environment/. But don’t completely follow it. You host doesn’t need to have a compositor, so only need to check whether CT started, if started, ssh to it, else start it then ssh to it. But how to run script when host started, either cron job or systemd, learn it yourself.
Basically, running a headless CT is almost the same as a headless VM, except autostart.