Modeswitch for unprivileged containers

I run a privileged container with X11 for local input and output on a proxmox host by mapping /dev/input, /dev/dri, and /dev/snd. This is working but for security reasons I would prefer unprivileged LXC with X11 running inside the container. My understanding from experiments is that the main issue is that modeswitch requires privileges, that are not available to the unprivileged container. Is it possible to develop a small utility on the host that switches from console to graphics and then hands over control of the graphics to the unprivileged container. How would that work?