Incus DE experience

, , ,
1

If Incus was to become a virtualisation technology for Linux desktops what would the user experience be?

Clearly Incus is designed for server deployments into datacentres for building cloud infrastructure. However there is a growing interest in the use of Incus to virtualise independent desktops and applications.

The standard graphical experience of Incus on a Linux desktop environment is similar to VMware Workstation — view the container or vm in a single window.

Could an Incus container or vm be experienced as a virtual workspace?

Wayland could be the way to do this with the strongest isolation and security.

The obvious benefit is the seemless support of multiple monitors and the ability to fluidly switch from host desktop environment to guest.

Ideally this would be possible through a shared virtual GPU and a dedicated passthrough GPU. Both providing full native acceleration to all environments.

Being able to route a guest’s display output from a dedicated passthrough GPU to a host workspace would be awesome, too.

It seems like Linux desktop is on the verge of brilliance but can desktop virtualisation achieve this level of functionality and usability?

2

Looking Glass might be the technology that enables part of this solution:
https://looking-glass.io/

Currently it is dedicated to, ugh, dedicated GPU passthrough from a guest OS to the host.

AMD just moved its Linux Radeon software over to the open source Vulkan drivers and maybe they will contribute to that codebase now.

Maybe we might see the likes of Distrobox getting on board Incus and perhaps contributing?

Everything in the clear and open source.

3

I’ve done GUI passthrough to system container instances as well as doing full GPU passthrough to virtual machines.

In the case of the former, it is possible to display graphical applications in an instance using the hosts wayland socket. A caveat however, you will also need to pass in X11 sockets as it doesn’t seem like, at least for fedora/42 and debian/13, Xwayland sockets are created in the instance.

Without passing through the host X11 sockets, programs like Visual Studio Code fail to start, although I’m not sure if this is representative of all Electron based programs.

There is also some theme-ing issues, specifically with KDE applications, I’ve been able to fix this in Debian based instances with a theme package - the name of which elludes me at the moment - but I’ve been less fortunate on RHEL based instances. I’ll check my current instances and report back with an edit when I can.

You can refer to these two links for more details: Incus / LXD profile for GUI apps: Wayland, X11 and Pulseaudio and 🌳 Running Graphical Applications in Incus Containers - Regrow.Earth

In the case of the latter, then there are actually two approaches here. The first is paravirtualized graphics, which would be: virtio-gpu-gl-pci [1] and Virtio-GPU Venus [1]. The two methods proxy OpenGL and Vulkan commands respectively to your GPU so you can get full 3D acceleration without the need of dedicating a GPU to your virtual machine.

Refer to this Collabora article for more details: The state of GFX virtualization using virglrenderer

And these two videos for a demonstration using games: https://www.youtube.com/watch?v=HmyQqrS09eo and https://www.youtube.com/watch?v=Vk6ux08UDuA&t=47s

That said, Virtio-GPU GL (same as virtio-gpu-gl) is the current default in Incus, Venus will require some configuration changes using raw.qemu.conf but it shouldn’t be too difficult to add in; the most concerning part, at least for me, is that I have been unable to replicate the performance in the listed demonstrations. Outside of these two, there is also Virtio-GPU Native Context, but you can refer to the Collabora article listed above for more details.

All of the above however requires running the latest kernel (6.14 at the minimum), latest Virglrenderer (v1.0.0 at the minimum [1]) and Qemu 9.2 and newer.

EDIT: as a follow-up to Virtio Native Context, this MR has been upstreamed for Native Context support for AMD GPUs: Making sure you're not a bot!, iirc Intel is either under review or has also been upstreamed. Qemu support for this will require a patch that is floating around in their mailing list, not sure on its status.

EDIT2: the patch in question: Making sure you're not a bot!

[1] virtio-gpu — QEMU documentation

The second approach is the generic, bog-standard, PCIe Passthrough using VFIO. As alluded to above this requires the passthrough of a GPU. If you’re usecase is having both the host and guest VM running in parallel then this option is a non-starter if you do not have two or more GPUs (not a major concern on AMD Zen 4 and newer and any Intel CPU without the F suffix).

2 Likes
4

Looking Glass might be the technology that enables part of this solution:
https://looking-glass.io/

Currently it is dedicated to, ugh, dedicated GPU passthrough from a guest OS to the host.

I’m fairly certain that Looking Glass is limited to Windows guest operating systems only; technically there is a Linux capture method, but it’s currently on the back burner.

AMD just moved its Linux Radeon software over to the open source Vulkan drivers and maybe they will contribute to that codebase now.

This part doesn’t seem too relevant, AMD’s KMD for their graphics cards has been upstreamed for a while already and their UMD has always seen more involvement both community-wise and commercially in the Mesa stack.

Maybe we might see the likes of Distrobox getting on board Incus and perhaps contributing?

This one seems very unlikely. Despite my own love for Incus, the majority of the Linux container ecosystem seems to have already settled in on OCI and Docker/Podman.

EDIT: that being said, it isn’t too difficult to replicate what Distrobox does within Incus instances.

1 Like
5

Great response thanks. You’ve done a lot of work in this area and these options and links are excellent. Yeah, Looking Glass is Windows guest only for now and they only support OpenGL (which is odd considering Vulkan has a much larger feature set).

Maybe someday workspace virtualisation will become a first class experience in Linux and available through Incus (on Wayland). Certainly there are some interesting projects that could move the state of the art in that direction and contribute ideas and code to each other.

QubesOS has attempted a virtualised desktop environment but it seems to be very niche in the highly secure category, which is a desirable objective and should of course be a first class citizen.

I want to run AI in a dev environment that isn’t a clunky grind to use every day. The AI must be fully sandboxed in a secure container. Distrobox isn’t an option. Just using Flatpak isn’t an option.

The main thing that will liberate a virtualised desktop is I think going to be some experience like guest display into a virtual workspace on the host. Support multiple monitors and many apps open across all displays.

And a well presented GUI experience for moving files and data between the virtualised environments such as an integrated authorisation GUI that adds to effortless flow.

Cheers

6

Maybe someday workspace virtualisation will become a first class experience in Linux and available through Incus (on Wayland). Certainly there are some interesting projects that could move the state of the art in that direction and contribute ideas and code to each other.

QubesOS has attempted a virtualised desktop environment but it seems to be very niche in the highly secure category, which is a desirable objective and should of course be a first class citizen.

In this regards, this was a blog post from a little while ago that could be retrofitted to suit your needs: Qubes-lite with KVM and Wayland - Thomas Leonard's blog. Although this is perhaps even more clunky than just using an Incus instance with GUI passthrough setup.

I also know that the QubesOS team is looking into transition into Wayland so maybe what they have done could also be applicable.

I want to run AI in a dev environment that isn’t a clunky grind to use every day. The AI must be fully sandboxed in a secure container. Distrobox isn’t an option. Just using Flatpak isn’t an option.

The main thing that will liberate a virtualised desktop is I think going to be some experience like guest display into a virtual workspace on the host. Support multiple monitors and many apps open across all displays.

And a well presented GUI experience for moving files and data between the virtualised environments such as an integrated authorisation GUI that adds to effortless flow.

Could you elaborate on your development environment stack? Maybe what you want to do is already possible with some more details. And if I’m unable to help, there’s probably someone else on this forum who could.

1 Like
7

You only need a tty to show container desktop environment in host:

But I didn’t test it when host has a desktop environment. And I didn’t test wayland. It’s all trial and error, no documents exist, try it yourself.

8

Thanks. I’m looking at Zed Editor with probably an AI like Claude, CodeX, Copilot. For Elm and Python, maybe some Fsharp. Pretty lightweight.

The machine is not lightweight however: Epyc ROME 32-cores with 384GB DDR4 and 4 x fast SSDs direct to CPU PCIe.

Running Fedora Silverblue 42. It has been the most stable so far. Some weirdness with display glitches but that’s probably Wayland and some of the apps.

I’d like to explore NixOS in a container.

The great thing about Fedora Atomic is it can be rebased into any variant including Universal Blue images.

Also running this stack will be some network virtualisation, likely Netlab. A DevOps Python based environment using Vagrant and QMEU.

With all this going on I want to keep the base OS as clean as possible while being as functional and fluid as possible.

9

Nothing immediately comes out as too clunky for daily use in an Incus system container instance, but you’ll need to do some upfront work.

Probably the first place I’d start off with is following the links for allowing GUI apps in instances.

Some apps don’t work well if launched as root, so you may want to create a non-root user prior to following those links. If you create a new instance then can take a look at cloud-init support in the Incus documentation [1] and the cloud-init modules reference [2].

[1] How to use cloud-init - Incus documentation

[2] Module reference - cloud-init 25.1.2 documentation

For Zed you’d probably want to create a .desktop entry file to launch the app from your host start menu. The exec command would be along the lines of: incus exec instance -- su -l user -c zed

See these links for more details:

For the command line applications you could create a profile in your terminal so it starts directly in an instance after which you can invoke python3 and so on. Or you could create bash aliases to do the same.

I don’t use Netlab so I can’t help you there. For Qemu, you’ll want to pass in /dev/kvm for KVM acceleration, and if you need other devices then you can pass it in to the instance with the respective device types listed in the documentation [3]

[3] Devices - Incus documentation

1 Like
10

Thanks, lots of good guidance there :pray: Good to hear that Incus can be a smooth platform for daily desktop/workstation use.

I’m going to spend the next week going through your recommendations for getting a dev environment up in Incus. Cheers.

11

The Incus Profile config for GPU virtualisation looks promising but I think I will go through manual setup and tune the installation until I can create configs. Probably the best breakdown of the Profile is the Regrow Earth source.

What would be awesome is a repo created under Zabbly for VDI and Desktop GPU virtualisation based on both Profile and Cloud-init config solutions (and whatever else is required). Something that becomes verified and persists at the official level instead of in forums and blogs all over the internet. The repo can then become a valuable reference architecture.

In my case I will be trying the pure Wayland/Vulkan pathway. No X11 and no OpenGL. My interest in Atomic distros also rules out Debian, Ubuntu and Mint. I liked Mint’s choice of components and design of their Cinnamon DE but no first class Wayland or Atomic rules them out.

12

That said, Virtio-GPU GL (same as virtio-gpu-gl ) is the current default in Incus

Is this correct? I’m searching on how to get virgl working on Incus vms and came across this post but it seems default gpu is always a virtio-vga device, so just basic 2D accel.

13

You can try using raw.qemu.conf to override the driver to virtio-gpu-gl-pci but last I looked into VirGL, it really wasn’t particularly suitable for Incus’ needs.

What I remember of how it worked was that it would effectively use the host’s OpenGL stack and interface it with the guest’s so that direct rendering requests for 3D content would bypass the virtual GPU entirely and hit the real GPU, having it do the drawing.

For that to work you need to have the VM be on the same system that’s running the VGA client and you need to know the geometry details of that client to know where to render things on screen.

With Incus being completely designed around a client/server concept, this wasn’t really working at all.

Anyway, that’s what I remember from some basic investigation I did a few years ago. If there’s a way to have 3D accelerated rendering inside the guest that’s still captured by the normal virtual GPU driver stack and gets fed over to SPICE, then that’d be a nice performance improvement for sure.

1 Like
14

Hey thanks for answering quickly! I could make both virgl and venus work in a 24.04 Ubuntu Incus VM, but only with the egl-headless backend, which worked but is a bad experience compared to gtk/sdl backends (it stalls occasionally, wont ramp power).

  raw.qemu: -display egl-headless,gl=on
  raw.qemu.conf: |-
    [device "qemu_gpu"]
    driver = "virtio-gpu-gl-pci"
    blob = "true"
    venus = "true"
    hostmem = "4G"

Next option to explore is trying non-venus virgl with spice, as libvirt does which is a bit better than egl-headless.

gtk/sdl backends I couldn’t make work sadly with incus, as they’re the most performant nowadays on qemu. Below is the qemu setup I use to test gtk backend.

qemu-system-x86_64                                           \
-enable-kvm                                                  \
-M q35                                                       \
-smp cpus=8,cores=4,threads=2,sockets=1                      \
-m 32G                                                       \
-cpu host,+topoext                                           \
-device virtio-vga-gl,hostmem=4G,blob=true,venus=true        \
-vga none                                                    \
-display gtk,gl=on,show-cursor=on                            \
-usb -device usb-tablet                                      \
-object memory-backend-memfd,id=mem1,size=32G                \
-machine memory-backend=mem1                                 \
-cdrom ~/Downloads/Fedora-Workstation-Live-42-1.1.x86_64.iso \

Hopefully masters will eventually share better solutions.

1 Like
15

Yeah, gtk/sdl is definitely going to be a problem given how we interact with QEMU.

But if there’s some way to get the venus stuff working with SPICE, that’d be nice.
Having some way to provide accelerated rendering within the guest even if the framebuffer capture part isn’t accelerated would still be a nice improvement.

We’re not going to have folks playing competitive FPS games through the SPICE console any time soon (or likely ever), but if there are ways to better accelerate the guests and save a bunch of CPU in the process, that’d be nice.

1 Like
16

It maybe worth keeping an eye on how Proxmox is approaching this. If this person is indeed an employee of Proxmox then this comment highlights how they plan on replacing Spice in the long term: > I wonder if tools like Proxmox could use this as a more efficient alternative ... | Hacker News

That said, I’m almost at the point where I’ve written off Virtio-GPU GL and Venus off until Native Context lands in Qemu. The amount of headache I’ve had in trying to get Venus to work, even on a local machine, was astounding; neither the SDL or GTK display options resulted in satisfactory performance [1] nor accurate input.

For some background on the environment I was attempting to run it on: Fedora 42 KDE, Qemu 9.2, virglrenderer 1.1, Kernel 6.14 (this was necessary because of certain patches needed upstream related to KVM); machine was Gigabyte Aorus 5 KE4 (i7-12700H, RTX 3060, 64GB DDR4).

Currently I have higher hopes at this point that Intel’s Arc B580 doesn’t have SRIOV fused off on the silicon. And as of this week and the last, a PR for SRIOV support (to be more accurate, exposing a has_sriov flag) for Alder Lake (and iirc, Rocket Lake was the codename), Alchemist, and BMG (Battlemage) was sent to drm-next.

[1] as noted in this demo video by TrippleXC on Reddit: https://www.youtube.com/watch?v=Vk6ux08UDuA or this one https://www.youtube.com/watch?v=HmyQqrS09eo

If I get some time I’ll try to verify and correct my earlier claims that Incus was using virtio-gpu-gl (I swear I saw that under the Qemu conf in /run/incus) and see if I can’t get something working on a newer kernel.

EDIT: following up:

[device “qemu_gpu”]
driver = “virtio-vga”
bus = “qemu_pcie3”
addr = "00.0”
2 Likes
17

Ok, I got both regular virgl and venus “kinda” working with incus+spice:

  raw.qemu: -spice unix=on,port=0,disable-ticketing=on,image-compression=off,gl=on,rendernode=/dev/dri/by-path/pci-0000:03:00.0-render,seamless-migration=on,addr=/run/incus/v1/qemu.spice
  raw.qemu.conf: |-
    [device "qemu_gpu"]
    driver = "virtio-vga-gl"
    blob = "true"
    hostmem = "4G"

# for venus, add: venus = "true"
# driver = "virtio-gpu-gl-pci" or "virtio-gpu-gl" also should work with slight differences

As it is, it breaks incus console (incus redirects spice socket to a new socket it seems, besides other things). Not entirely sure why yet. So, I connected to /run/incus/v1/qemu.spice directly using spicy (spice client). remote-viewer should also work.

Performance on my AMD card (7900 XTX) seems decent enough for desktop usage, I tested in Fedora 42 and Ubuntu 24.04 GNOME Wayland, Firefox with a bunch of tabs, terminals and VSCode. Certainly a LOT better than virtio-vga, smoother and relatively low CPU usage even when dragging windows around.

Bad news is that incus probably would need changes or some more advanced config-fu for incus console to work.

1 Like
18

I’ve been expecting to build out a seamless host-centric desktop environment (in addition to learning how to build cloud infrastructure on Incus) but I only now realise and accept that fundamental design decisions in the creation of Incus (inherited from LXD) required to provide a true cloud-centeic client/server platform may limit desktop solutions. If not limit then slow the integration of emerging desktop solutions.

But perhaps some of the emerging tech will make it easy to develop desktop-centric integrations and features. But currently not.

I’ve also been focusing on containers and only recently considered virtual machines for desktop. Since then have learned more about QEMU and discovered `virt-manager` which seems to be a complete gui VMM for QMEU and KVM.

Considering that Venus, vDRM and Native Context appear to be targeted for and most compatible with QEMU’s VMM perhaps for a streamlined host-centric desktop experience I should use native QMEU instead of Incus?

Incus could then be run inside QMEU virtual machines for CLI only server OSes.

The downside is I’ll probably need to remove Incus on the host and reformat the SSD dedicated to an Incus zfs store. Host is Debian Trixie.

Zed Editor requires a good level of GPU hardware acceleration and my current Incus VM doesn’t provide this.

Thoughts?

19

Incus is a hypervisor platform that unifies the management of containers and virtual machines. Incus provides general facilities that can be used in useful ways by different consumers of the software.

About twenty years ago when there was interest to have some kernel support for something that would become what we know now as Linux containers. One option was to create some sort of subsystem for Linux containers. The kernel developers were against this, because a dedicated such subsystem is generally complicated, and if the container fad does not work out, then you are stuck in maintaining a whole subsystem. What really happened, and it is good design practice, is to add primitives to the Linux kernel ( namespaces, cgroups, etc) that can be used to build Linux containers. In a way, that subsystem would now be running in user-space.

By following the same design pattern, Incus has the facilities that are required to implement many things, even things not envisioned in the early days. As far as I know, there is nothing fundamental missing on the part of Incus to have a Desktop Environment experience.

There are ways to run individual GUI applications in a system container and get the output on the host’s desktop. In some cases it may not be that seamless, but that’s up to us to figure out what is needed to make it seamless.

There are also ways to run a Desktop Environment in a system container and a virtual machine. The latter is more straightforward, and any performance issues have to do with external-to-Incus matters. In a VM you either use a dedicated GPU, or you use some other software that sends the output to the host’s GPU (Xspice).

Running a Desktop Environment in a system container looks promising to me. @tarruda wrote the tutorial and I got it working in Incus. The Linux distribution runtimes that are used to generate the system container images, use stock packages. Those packages do not work as is, when they are running inside a system container. They require tiny changes that have not been upstreamed yet. If those changes get upstreamed, then it would be possible to use stock container images to launch desktop environments. Failing that, it’s just an issue of creating another Incus remote that will have such ready-made container images (created manually).

1 Like
20

Running a Desktop Environment in a system container looks promising to me. @tarruda wrote the tutorial and I got it working in Incus.

The problem with this approach is that it relies on X11 and Spice. The general trend, at least with Gnome, is to move away and deprecate the X11 components in favour of Wayland, and in that aspect, Wayland has not been a major priority for remoting software due to lack of protocol components. Gnome and KDE do ship a built-in RDP server however.

While Gnome has backtracked, the first proposals for Gnome 49 was the complete removal of X11 components in their DE.

It also remains to be seen how actively Spice will be maintained since Redhat deprecated it since 8.3. The mailing list archives don’t paint a great picture nor does their Gitlab activity. Even Proxmox is looking to migrate away from Spice: > I wonder if tools like Proxmox could use this as a more efficient alternative ... | Hacker News

It’s also as you said that the components are there, but the glue does not. You can pass through the host’s Wayland and X11 sockets to the guest but configuration of such is left as an exercise to the user. My current attempts have largely resulted in working Wayland and X11 programs but not theming integration; personal future exploration is to look at nested KDE compositor inside a system container.

I can’t speak for Gnome but KDE also has a documentation bit that shows how to run a nested KDE Wayland session. But not much regarding headless Wayland hosts.

@nickwalt

There are also misconceptions in your comment, Incus integrates Qemu into its stack much like how VMM (edit: VMM as in Virtual Machine Manager, the former Redhat project) does with Qemu. If you’re mainly interested in a GUI solution now, then you can take a look into Gnome Boxes (or the equivalent KDE project that eludes me at the moment.)

1 Like