Incus DE experience

, , ,
1

If Incus was to become a virtualisation technology for Linux desktops what would the user experience be?

Clearly Incus is designed for server deployments into datacentres for building cloud infrastructure. However there is a growing interest in the use of Incus to virtualise independent desktops and applications.

The standard graphical experience of Incus on a Linux desktop environment is similar to VMware Workstation — view the container or vm in a single window.

Could an Incus container or vm be experienced as a virtual workspace?

Wayland could be the way to do this with the strongest isolation and security.

The obvious benefit is the seemless support of multiple monitors and the ability to fluidly switch from host desktop environment to guest.

Ideally this would be possible through a shared virtual GPU and a dedicated passthrough GPU. Both providing full native acceleration to all environments.

Being able to route a guest’s display output from a dedicated passthrough GPU to a host workspace would be awesome, too.

It seems like Linux desktop is on the verge of brilliance but can desktop virtualisation achieve this level of functionality and usability?

2

Looking Glass might be the technology that enables part of this solution:
https://looking-glass.io/

Currently it is dedicated to, ugh, dedicated GPU passthrough from a guest OS to the host.

AMD just moved its Linux Radeon software over to the open source Vulkan drivers and maybe they will contribute to that codebase now.

Maybe we might see the likes of Distrobox getting on board Incus and perhaps contributing?

Everything in the clear and open source.

3

I’ve done GUI passthrough to system container instances as well as doing full GPU passthrough to virtual machines.

In the case of the former, it is possible to display graphical applications in an instance using the hosts wayland socket. A caveat however, you will also need to pass in X11 sockets as it doesn’t seem like, at least for fedora/42 and debian/13, Xwayland sockets are created in the instance.

Without passing through the host X11 sockets, programs like Visual Studio Code fail to start, although I’m not sure if this is representative of all Electron based programs.

There is also some theme-ing issues, specifically with KDE applications, I’ve been able to fix this in Debian based instances with a theme package - the name of which elludes me at the moment - but I’ve been less fortunate on RHEL based instances. I’ll check my current instances and report back with an edit when I can.

You can refer to these two links for more details: Incus / LXD profile for GUI apps: Wayland, X11 and Pulseaudio and 🌳 Running Graphical Applications in Incus Containers - Regrow.Earth

In the case of the latter, then there are actually two approaches here. The first is paravirtualized graphics, which would be: virtio-gpu-gl-pci [1] and Virtio-GPU Venus [1]. The two methods proxy OpenGL and Vulkan commands respectively to your GPU so you can get full 3D acceleration without the need of dedicating a GPU to your virtual machine.

Refer to this Collabora article for more details: The state of GFX virtualization using virglrenderer

And these two videos for a demonstration using games: https://www.youtube.com/watch?v=HmyQqrS09eo and https://www.youtube.com/watch?v=Vk6ux08UDuA&t=47s

That said, Virtio-GPU GL (same as virtio-gpu-gl) is the current default in Incus, Venus will require some configuration changes using raw.qemu.conf but it shouldn’t be too difficult to add in; the most concerning part, at least for me, is that I have been unable to replicate the performance in the listed demonstrations. Outside of these two, there is also Virtio-GPU Native Context, but you can refer to the Collabora article listed above for more details.

All of the above however requires running the latest kernel (6.14 at the minimum), latest Virglrenderer (v1.0.0 at the minimum [1]) and Qemu 9.2 and newer.

EDIT: as a follow-up to Virtio Native Context, this MR has been upstreamed for Native Context support for AMD GPUs: Making sure you're not a bot!, iirc Intel is either under review or has also been upstreamed. Qemu support for this will require a patch that is floating around in their mailing list, not sure on its status.

EDIT2: the patch in question: Making sure you're not a bot!

[1] virtio-gpu — QEMU documentation

The second approach is the generic, bog-standard, PCIe Passthrough using VFIO. As alluded to above this requires the passthrough of a GPU. If you’re usecase is having both the host and guest VM running in parallel then this option is a non-starter if you do not have two or more GPUs (not a major concern on AMD Zen 4 and newer and any Intel CPU without the F suffix).

2 Likes
4

Looking Glass might be the technology that enables part of this solution:
https://looking-glass.io/

Currently it is dedicated to, ugh, dedicated GPU passthrough from a guest OS to the host.

I’m fairly certain that Looking Glass is limited to Windows guest operating systems only; technically there is a Linux capture method, but it’s currently on the back burner.

AMD just moved its Linux Radeon software over to the open source Vulkan drivers and maybe they will contribute to that codebase now.

This part doesn’t seem too relevant, AMD’s KMD for their graphics cards has been upstreamed for a while already and their UMD has always seen more involvement both community-wise and commercially in the Mesa stack.

Maybe we might see the likes of Distrobox getting on board Incus and perhaps contributing?

This one seems very unlikely. Despite my own love for Incus, the majority of the Linux container ecosystem seems to have already settled in on OCI and Docker/Podman.

EDIT: that being said, it isn’t too difficult to replicate what Distrobox does within Incus instances.

1 Like
5

Great response thanks. You’ve done a lot of work in this area and these options and links are excellent. Yeah, Looking Glass is Windows guest only for now and they only support OpenGL (which is odd considering Vulkan has a much larger feature set).

Maybe someday workspace virtualisation will become a first class experience in Linux and available through Incus (on Wayland). Certainly there are some interesting projects that could move the state of the art in that direction and contribute ideas and code to each other.

QubesOS has attempted a virtualised desktop environment but it seems to be very niche in the highly secure category, which is a desirable objective and should of course be a first class citizen.

I want to run AI in a dev environment that isn’t a clunky grind to use every day. The AI must be fully sandboxed in a secure container. Distrobox isn’t an option. Just using Flatpak isn’t an option.

The main thing that will liberate a virtualised desktop is I think going to be some experience like guest display into a virtual workspace on the host. Support multiple monitors and many apps open across all displays.

And a well presented GUI experience for moving files and data between the virtualised environments such as an integrated authorisation GUI that adds to effortless flow.

Cheers

6

Maybe someday workspace virtualisation will become a first class experience in Linux and available through Incus (on Wayland). Certainly there are some interesting projects that could move the state of the art in that direction and contribute ideas and code to each other.

QubesOS has attempted a virtualised desktop environment but it seems to be very niche in the highly secure category, which is a desirable objective and should of course be a first class citizen.

In this regards, this was a blog post from a little while ago that could be retrofitted to suit your needs: Qubes-lite with KVM and Wayland - Thomas Leonard's blog. Although this is perhaps even more clunky than just using an Incus instance with GUI passthrough setup.

I also know that the QubesOS team is looking into transition into Wayland so maybe what they have done could also be applicable.

I want to run AI in a dev environment that isn’t a clunky grind to use every day. The AI must be fully sandboxed in a secure container. Distrobox isn’t an option. Just using Flatpak isn’t an option.

The main thing that will liberate a virtualised desktop is I think going to be some experience like guest display into a virtual workspace on the host. Support multiple monitors and many apps open across all displays.

And a well presented GUI experience for moving files and data between the virtualised environments such as an integrated authorisation GUI that adds to effortless flow.

Could you elaborate on your development environment stack? Maybe what you want to do is already possible with some more details. And if I’m unable to help, there’s probably someone else on this forum who could.

1 Like
7

You only need a tty to show container desktop environment in host:

But I didn’t test it when host has a desktop environment. And I didn’t test wayland. It’s all trial and error, no documents exist, try it yourself.

8

Thanks. I’m looking at Zed Editor with probably an AI like Claude, CodeX, Copilot. For Elm and Python, maybe some Fsharp. Pretty lightweight.

The machine is not lightweight however: Epyc ROME 32-cores with 384GB DDR4 and 4 x fast SSDs direct to CPU PCIe.

Running Fedora Silverblue 42. It has been the most stable so far. Some weirdness with display glitches but that’s probably Wayland and some of the apps.

I’d like to explore NixOS in a container.

The great thing about Fedora Atomic is it can be rebased into any variant including Universal Blue images.

Also running this stack will be some network virtualisation, likely Netlab. A DevOps Python based environment using Vagrant and QMEU.

With all this going on I want to keep the base OS as clean as possible while being as functional and fluid as possible.

9

Nothing immediately comes out as too clunky for daily use in an Incus system container instance, but you’ll need to do some upfront work.

Probably the first place I’d start off with is following the links for allowing GUI apps in instances.

Some apps don’t work well if launched as root, so you may want to create a non-root user prior to following those links. If you create a new instance then can take a look at cloud-init support in the Incus documentation [1] and the cloud-init modules reference [2].

[1] How to use cloud-init - Incus documentation

[2] Module reference - cloud-init 25.1.2 documentation

For Zed you’d probably want to create a .desktop entry file to launch the app from your host start menu. The exec command would be along the lines of: incus exec instance -- su -l user -c zed

See these links for more details:

For the command line applications you could create a profile in your terminal so it starts directly in an instance after which you can invoke python3 and so on. Or you could create bash aliases to do the same.

I don’t use Netlab so I can’t help you there. For Qemu, you’ll want to pass in /dev/kvm for KVM acceleration, and if you need other devices then you can pass it in to the instance with the respective device types listed in the documentation [3]

[3] Devices - Incus documentation

1 Like
10

Thanks, lots of good guidance there :pray: Good to hear that Incus can be a smooth platform for daily desktop/workstation use.

I’m going to spend the next week going through your recommendations for getting a dev environment up in Incus. Cheers.

11

The Incus Profile config for GPU virtualisation looks promising but I think I will go through manual setup and tune the installation until I can create configs. Probably the best breakdown of the Profile is the Regrow Earth source.

What would be awesome is a repo created under Zabbly for VDI and Desktop GPU virtualisation based on both Profile and Cloud-init config solutions (and whatever else is required). Something that becomes verified and persists at the official level instead of in forums and blogs all over the internet. The repo can then become a valuable reference architecture.

In my case I will be trying the pure Wayland/Vulkan pathway. No X11 and no OpenGL. My interest in Atomic distros also rules out Debian, Ubuntu and Mint. I liked Mint’s choice of components and design of their Cinnamon DE but no first class Wayland or Atomic rules them out.

12

That said, Virtio-GPU GL (same as virtio-gpu-gl ) is the current default in Incus

Is this correct? I’m searching on how to get virgl working on Incus vms and came across this post but it seems default gpu is always a virtio-vga device, so just basic 2D accel.

13

You can try using raw.qemu.conf to override the driver to virtio-gpu-gl-pci but last I looked into VirGL, it really wasn’t particularly suitable for Incus’ needs.

What I remember of how it worked was that it would effectively use the host’s OpenGL stack and interface it with the guest’s so that direct rendering requests for 3D content would bypass the virtual GPU entirely and hit the real GPU, having it do the drawing.

For that to work you need to have the VM be on the same system that’s running the VGA client and you need to know the geometry details of that client to know where to render things on screen.

With Incus being completely designed around a client/server concept, this wasn’t really working at all.

Anyway, that’s what I remember from some basic investigation I did a few years ago. If there’s a way to have 3D accelerated rendering inside the guest that’s still captured by the normal virtual GPU driver stack and gets fed over to SPICE, then that’d be a nice performance improvement for sure.

14

Hey thanks for answering quickly! I could make both virgl and venus work in a 24.04 Ubuntu Incus VM, but only with the egl-headless backend, which worked but is a bad experience compared to gtk/sdl backends (it stalls occasionally, wont ramp power).

  raw.qemu: -display egl-headless,gl=on
  raw.qemu.conf: |-
    [device "qemu_gpu"]
    driver = "virtio-gpu-gl-pci"
    blob = "true"
    venus = "true"
    hostmem = "4G"

Next option to explore is trying non-venus virgl with spice, as libvirt does which is a bit better than egl-headless.

gtk/sdl backends I couldn’t make work sadly with incus, as they’re the most performant nowadays on qemu. Below is the qemu setup I use to test gtk backend.

qemu-system-x86_64                                           \
-enable-kvm                                                  \
-M q35                                                       \
-smp cpus=8,cores=4,threads=2,sockets=1                      \
-m 32G                                                       \
-cpu host,+topoext                                           \
-device virtio-vga-gl,hostmem=4G,blob=true,venus=true        \
-vga none                                                    \
-display gtk,gl=on,show-cursor=on                            \
-usb -device usb-tablet                                      \
-object memory-backend-memfd,id=mem1,size=32G                \
-machine memory-backend=mem1                                 \
-cdrom ~/Downloads/Fedora-Workstation-Live-42-1.1.x86_64.iso \

Hopefully masters will eventually share better solutions.

15

Yeah, gtk/sdl is definitely going to be a problem given how we interact with QEMU.

But if there’s some way to get the venus stuff working with SPICE, that’d be nice.
Having some way to provide accelerated rendering within the guest even if the framebuffer capture part isn’t accelerated would still be a nice improvement.

We’re not going to have folks playing competitive FPS games through the SPICE console any time soon (or likely ever), but if there are ways to better accelerate the guests and save a bunch of CPU in the process, that’d be nice.

16

It maybe worth keeping an eye on how Proxmox is approaching this. If this person is indeed an employee of Proxmox then this comment highlights how they plan on replacing Spice in the long term: > I wonder if tools like Proxmox could use this as a more efficient alternative ... | Hacker News

That said, I’m almost at the point where I’ve written off Virtio-GPU GL and Venus off until Native Context lands in Qemu. The amount of headache I’ve had in trying to get Venus to work, even on a local machine, was astounding; neither the SDL or GTK display options resulted in satisfactory performance [1] nor accurate input.

For some background on the environment I was attempting to run it on: Fedora 42 KDE, Qemu 9.2, virglrenderer 1.1, Kernel 6.14 (this was necessary because of certain patches needed upstream related to KVM); machine was Gigabyte Aorus 5 KE4 (i7-12700H, RTX 3060, 64GB DDR4).

Currently I have higher hopes at this point that Intel’s Arc B580 doesn’t have SRIOV fused off on the silicon. And as of this week and the last, a PR for SRIOV support (to be more accurate, exposing a has_sriov flag) for Alder Lake (and iirc, Rocket Lake was the codename), Alchemist, and BMG (Battlemage) was sent to drm-next.

[1] as noted in this demo video by TrippleXC on Reddit: https://www.youtube.com/watch?v=Vk6ux08UDuA or this one https://www.youtube.com/watch?v=HmyQqrS09eo

If I get some time I’ll try to verify and correct my earlier claims that Incus was using virtio-gpu-gl (I swear I saw that under the Qemu conf in /run/incus) and see if I can’t get something working on a newer kernel.

1 Like
17

Ok, I got both regular virgl and venus “kinda” working with incus+spice:

  raw.qemu: -spice unix=on,port=0,disable-ticketing=on,image-compression=off,gl=on,rendernode=/dev/dri/by-path/pci-0000:03:00.0-render,seamless-migration=on,addr=/run/incus/v1/qemu.spice
  raw.qemu.conf: |-
    [device "qemu_gpu"]
    driver = "virtio-vga-gl"
    blob = "true"
    hostmem = "4G"

# for venus, add: venus = "true"
# driver = "virtio-gpu-gl-pci" or "virtio-gpu-gl" also should work with slight differences

As it is, it breaks incus console (incus redirects spice socket to a new socket it seems, besides other things). Not entirely sure why yet. So, I connected to /run/incus/v1/qemu.spice directly using spicy (spice client). remote-viewer should also work.

Performance on my AMD card (7900 XTX) seems decent enough for desktop usage, I tested in Fedora 42 and Ubuntu 24.04 GNOME Wayland, Firefox with a bunch of tabs, terminals and VSCode. Certainly a LOT better than virtio-vga, smoother and relatively low CPU usage even when dragging windows around.

Bad news is that incus probably would need changes or some more advanced config-fu for incus console to work.