Incus DE experience

, , ,
1

If Incus was to become a virtualisation technology for Linux desktops what would the user experience be?

Clearly Incus is designed for server deployments into datacentres for building cloud infrastructure. However there is a growing interest in the use of Incus to virtualise independent desktops and applications.

The standard graphical experience of Incus on a Linux desktop environment is similar to VMware Workstation — view the container or vm in a single window.

Could an Incus container or vm be experienced as a virtual workspace?

Wayland could be the way to do this with the strongest isolation and security.

The obvious benefit is the seemless support of multiple monitors and the ability to fluidly switch from host desktop environment to guest.

Ideally this would be possible through a shared virtual GPU and a dedicated passthrough GPU. Both providing full native acceleration to all environments.

Being able to route a guest’s display output from a dedicated passthrough GPU to a host workspace would be awesome, too.

It seems like Linux desktop is on the verge of brilliance but can desktop virtualisation achieve this level of functionality and usability?

2

Looking Glass might be the technology that enables part of this solution:
https://looking-glass.io/

Currently it is dedicated to, ugh, dedicated GPU passthrough from a guest OS to the host.

AMD just moved its Linux Radeon software over to the open source Vulkan drivers and maybe they will contribute to that codebase now.

Maybe we might see the likes of Distrobox getting on board Incus and perhaps contributing?

Everything in the clear and open source.

3

I’ve done GUI passthrough to system container instances as well as doing full GPU passthrough to virtual machines.

In the case of the former, it is possible to display graphical applications in an instance using the hosts wayland socket. A caveat however, you will also need to pass in X11 sockets as it doesn’t seem like, at least for fedora/42 and debian/13, Xwayland sockets are created in the instance.

Without passing through the host X11 sockets, programs like Visual Studio Code fail to start, although I’m not sure if this is representative of all Electron based programs.

There is also some theme-ing issues, specifically with KDE applications, I’ve been able to fix this in Debian based instances with a theme package - the name of which elludes me at the moment - but I’ve been less fortunate on RHEL based instances. I’ll check my current instances and report back with an edit when I can.

You can refer to these two links for more details: Incus / LXD profile for GUI apps: Wayland, X11 and Pulseaudio and 🌳 Running Graphical Applications in Incus Containers - Regrow.Earth

In the case of the latter, then there are actually two approaches here. The first is paravirtualized graphics, which would be: virtio-gpu-gl-pci [1] and Virtio-GPU Venus [1]. The two methods proxy OpenGL and Vulkan commands respectively to your GPU so you can get full 3D acceleration without the need of dedicating a GPU to your virtual machine.

Refer to this Collabora article for more details: The state of GFX virtualization using virglrenderer

And these two videos for a demonstration using games: https://www.youtube.com/watch?v=HmyQqrS09eo and https://www.youtube.com/watch?v=Vk6ux08UDuA&t=47s

That said, Virtio-GPU GL (same as virtio-gpu-gl) is the current default in Incus, Venus will require some configuration changes using raw.qemu.conf but it shouldn’t be too difficult to add in; the most concerning part, at least for me, is that I have been unable to replicate the performance in the listed demonstrations. Outside of these two, there is also Virtio-GPU Native Context, but you can refer to the Collabora article listed above for more details.

All of the above however requires running the latest kernel (6.14 at the minimum), latest Virglrenderer (v1.0.0 at the minimum [1]) and Qemu 9.2 and newer.

EDIT: as a follow-up to Virtio Native Context, this MR has been upstreamed for Native Context support for AMD GPUs: Making sure you're not a bot!, iirc Intel is either under review or has also been upstreamed. Qemu support for this will require a patch that is floating around in their mailing list, not sure on its status.

EDIT2: the patch in question: Making sure you're not a bot!

[1] virtio-gpu — QEMU documentation

The second approach is the generic, bog-standard, PCIe Passthrough using VFIO. As alluded to above this requires the passthrough of a GPU. If you’re usecase is having both the host and guest VM running in parallel then this option is a non-starter if you do not have two or more GPUs (not a major concern on AMD Zen 4 and newer and any Intel CPU without the F suffix).

2 Likes
4

Looking Glass might be the technology that enables part of this solution:
https://looking-glass.io/

Currently it is dedicated to, ugh, dedicated GPU passthrough from a guest OS to the host.

I’m fairly certain that Looking Glass is limited to Windows guest operating systems only; technically there is a Linux capture method, but it’s currently on the back burner.

AMD just moved its Linux Radeon software over to the open source Vulkan drivers and maybe they will contribute to that codebase now.

This part doesn’t seem too relevant, AMD’s KMD for their graphics cards has been upstreamed for a while already and their UMD has always seen more involvement both community-wise and commercially in the Mesa stack.

Maybe we might see the likes of Distrobox getting on board Incus and perhaps contributing?

This one seems very unlikely. Despite my own love for Incus, the majority of the Linux container ecosystem seems to have already settled in on OCI and Docker/Podman.

EDIT: that being said, it isn’t too difficult to replicate what Distrobox does within Incus instances.

1 Like
5

Great response thanks. You’ve done a lot of work in this area and these options and links are excellent. Yeah, Looking Glass is Windows guest only for now and they only support OpenGL (which is odd considering Vulkan has a much larger feature set).

Maybe someday workspace virtualisation will become a first class experience in Linux and available through Incus (on Wayland). Certainly there are some interesting projects that could move the state of the art in that direction and contribute ideas and code to each other.

QubesOS has attempted a virtualised desktop environment but it seems to be very niche in the highly secure category, which is a desirable objective and should of course be a first class citizen.

I want to run AI in a dev environment that isn’t a clunky grind to use every day. The AI must be fully sandboxed in a secure container. Distrobox isn’t an option. Just using Flatpak isn’t an option.

The main thing that will liberate a virtualised desktop is I think going to be some experience like guest display into a virtual workspace on the host. Support multiple monitors and many apps open across all displays.

And a well presented GUI experience for moving files and data between the virtualised environments such as an integrated authorisation GUI that adds to effortless flow.

Cheers

6

Maybe someday workspace virtualisation will become a first class experience in Linux and available through Incus (on Wayland). Certainly there are some interesting projects that could move the state of the art in that direction and contribute ideas and code to each other.

QubesOS has attempted a virtualised desktop environment but it seems to be very niche in the highly secure category, which is a desirable objective and should of course be a first class citizen.

In this regards, this was a blog post from a little while ago that could be retrofitted to suit your needs: Qubes-lite with KVM and Wayland - Thomas Leonard's blog. Although this is perhaps even more clunky than just using an Incus instance with GUI passthrough setup.

I also know that the QubesOS team is looking into transition into Wayland so maybe what they have done could also be applicable.

I want to run AI in a dev environment that isn’t a clunky grind to use every day. The AI must be fully sandboxed in a secure container. Distrobox isn’t an option. Just using Flatpak isn’t an option.

The main thing that will liberate a virtualised desktop is I think going to be some experience like guest display into a virtual workspace on the host. Support multiple monitors and many apps open across all displays.

And a well presented GUI experience for moving files and data between the virtualised environments such as an integrated authorisation GUI that adds to effortless flow.

Could you elaborate on your development environment stack? Maybe what you want to do is already possible with some more details. And if I’m unable to help, there’s probably someone else on this forum who could.

1 Like
7

You only need a tty to show container desktop environment in host:

But I didn’t test it when host has a desktop environment. And I didn’t test wayland. It’s all trial and error, no documents exist, try it yourself.

8

Thanks. I’m looking at Zed Editor with probably an AI like Claude, CodeX, Copilot. For Elm and Python, maybe some Fsharp. Pretty lightweight.

The machine is not lightweight however: Epyc ROME 32-cores with 384GB DDR4 and 4 x fast SSDs direct to CPU PCIe.

Running Fedora Silverblue 42. It has been the most stable so far. Some weirdness with display glitches but that’s probably Wayland and some of the apps.

I’d like to explore NixOS in a container.

The great thing about Fedora Atomic is it can be rebased into any variant including Universal Blue images.

Also running this stack will be some network virtualisation, likely Netlab. A DevOps Python based environment using Vagrant and QMEU.

With all this going on I want to keep the base OS as clean as possible while being as functional and fluid as possible.

9

Nothing immediately comes out as too clunky for daily use in an Incus system container instance, but you’ll need to do some upfront work.

Probably the first place I’d start off with is following the links for allowing GUI apps in instances.

Some apps don’t work well if launched as root, so you may want to create a non-root user prior to following those links. If you create a new instance then can take a look at cloud-init support in the Incus documentation [1] and the cloud-init modules reference [2].

[1] How to use cloud-init - Incus documentation

[2] Module reference - cloud-init 25.1.2 documentation

For Zed you’d probably want to create a .desktop entry file to launch the app from your host start menu. The exec command would be along the lines of: incus exec instance -- su -l user -c zed

See these links for more details:

For the command line applications you could create a profile in your terminal so it starts directly in an instance after which you can invoke python3 and so on. Or you could create bash aliases to do the same.

I don’t use Netlab so I can’t help you there. For Qemu, you’ll want to pass in /dev/kvm for KVM acceleration, and if you need other devices then you can pass it in to the instance with the respective device types listed in the documentation [3]

[3] Devices - Incus documentation

1 Like
10

Thanks, lots of good guidance there :pray: Good to hear that Incus can be a smooth platform for daily desktop/workstation use.

I’m going to spend the next week going through your recommendations for getting a dev environment up in Incus. Cheers.

11

The Incus Profile config for GPU virtualisation looks promising but I think I will go through manual setup and tune the installation until I can create configs. Probably the best breakdown of the Profile is the Regrow Earth source.

What would be awesome is a repo created under Zabbly for VDI and Desktop GPU virtualisation based on both Profile and Cloud-init config solutions (and whatever else is required). Something that becomes verified and persists at the official level instead of in forums and blogs all over the internet. The repo can then become a valuable reference architecture.

In my case I will be trying the pure Wayland/Vulkan pathway. No X11 and no OpenGL. My interest in Atomic distros also rules out Debian, Ubuntu and Mint. I liked Mint’s choice of components and design of their Cinnamon DE but no first class Wayland or Atomic rules them out.