Incus DE experience

, , ,
21

It makes sense that people might be moving away from SPICE because from what I read there is a VNC component in the protocol which displays guests. If SPICE is on its way out what is replacing it for QEMU display and IO integration?

I understood that Incus integrates into QEMU but from what I saw Incus runs its own VMM which doesn’t yet have the integration and support for the emerging Venus, vDRM and Native Context virtio-gpu technologies to the extent that the native QEMU VMM does. Why? Because that has not been its primary mission and I think @stgraber said it very clearly. Strangely it has taken me this long to get it.

Not sure where Rutabaga, gfxstream and Wayland display passthrough fits. Or the relevance, to Linux virtualisation based on QEMU-KVM, of Google’s Crosvm. Maybe there is a (understandable) reluctance to adopt Google “Open Source” code (if not the project itself). Intel’s Cloud Hypervisor technology is still in very active development and borrows from the other corporate virtualisation projects Firecracker and Crosvm. There are references to Crosvm regarding GPU hardware passthrough but it looks like it is a replacement or alternative to QEMU for integrating and managing KVM workloads.

Thoughts?

Looked at Gnome Boxes and it is a nice project. Could not find anything about their GPU hardware acceleration or support for Wayland on the host and in guests.

22

There is a lack of proper academic definition of the difference between a hypervisor and a virtual machine manager/monitor, but if we take VMware’s word for it they are synonymous.

Even if they were not, then Qemu would still be the hypervisor/VMM. Incus doesn’t do anything low-level except for configuring and invoking Qemu; so in this aspect, the Virtual Machine Manager (the old Redhat project) is not necessary as Incus is already synonymous to it

Virtio-GPU frankly is still a very new domain. Virtio-GPU Venus was only recently upstreamed in both Qemu, Mesa, Linux, and virglrenderer and so actual widespread use is going to be very low. The next contender is Virito-GPU Native Context (or Virito vDRM and so on… Collabora has an article with a comment clarifying that they are one the same) which is still in the process of being upstreamed.

In terms of using it in Incus, yes you’ll have to fiddle around with `raw.qemu.conf` but configuration should be relatively straight forward except for actually remoting in or using the standard desktop display windows (SDL or GTK)

Rutabaga is something I know little about, only that it is a Google project and seems to be heavily tied into Android; it’s possible this is used in their ChromeOS-Android merger but I cannot say. It is also less so reluctance, but more so that Google technologies are proprietary despite being FOSS. No one understands it best other than Google and realistically they are doing the lion’s share of the contributions.

Cloud Hypervisor, Firecracker, and CrosVM all share the commonality of being based on KVM and Virtio paravirtualization, same as Qemu. The difference in features is left to you as an exercise but you will find that, for now, Qemu is by far the most mature and still the attention of industry.

CrosVM’s method of GPU passthrough is something I’m not intimately familiar with but have interacted with; it is the basis of their Linux Environment feature on Chromebooks and for their Android emulator. I presume that the aforementioned Rutabaga may have a role in this area but I cannot say definitively.

Gnome Boxes uses libvirt underneath so what you can do with plain Virtual Machine Manager you can do in here, but that is a domain I know nothing about.

1 Like
23

Regarding Spice, frankly I don’t know. The realm of remote desktop is honestly quite mystifying with no clear obvious route. There is RustDesk but they have their share of concerns such as needing to disable Wayland in the past and, correct me if I am wrong, needing custom drivers with unsigned certificates for Windows.

There is TigerVNC but they are stuck on X11 until Wayland merges specific remote desktop extensions.

There is RDP (as in the Microsoft RDP) but this has to be handled in the Wayland compositor. Gnome and KDE ship a server that you can enable in their respective settings program.

1 Like
24

This talk segment from a Google engineer puts a few things into context and highlights Virtgpu DRM Native Context as the likely technology to dominate:

25

Yeah Native Context is by far the most interesting technology due to passing direct GPU commands over virito (with modifications to the kernel and user mode graphics drivers) compared to Viritio-GPU GL and Venus passing high level OpenGL or Vulkan commands.

I think the Asahi Linux is the first (or at least among the first) to demonstrate by playing Steam games. Side note: but the Asahi Linux team had to do this because of problems relating to the 16KiB host page size while they can emulate it as 4KiB in VMs.

I currently cannot find the link via Google but Intel should also have or is close to being upstream for their Virtio-GPU Native Context modifications.

I think Qemu is in their 13th round for upstreaming for Virtio GPU Native Context support.

1 Like
26

Cheers, by “…using the standard desktop display windows (SDL or GTK).” are you saying Virtio-GPU Venus has problems rendering GTK windows?

Interesting news about Native Context.

27

No I should have been more clear on that.

So for Incus actually displaying a window is problematic because of the hub and spoke or client-server model. Incus doesn’t run the VM under your user (at least if I remember correctly) so it lacks the functionality to display anything on the user’s Wayland or X11 session.

Spice doesn’t work, or at least it didn’t for me when I tried using Venus externally of Incus, due to Qemu not permitting it. Not sure how things have changed in this regard, seeing as someone earlier got it working.

Now outside of Incus, Qemu has had problems in my experience handling Venus acceleration (you can read my earlier posts), but this is likely a problem on my end seeing how others’ setups are working flawlessly.

28

As a follow-up to a point I made regarding Asahi Linux, you can read their native context efforts here: Beyond Gaming: X11 bridging in muvm - Asahi Linux

29

I’ve been able to run Spice integrated with Incus no problem using:
incus console <vm_name> –type=vga

The only problem is the rendering is in software. Not sure if it is using x11 or Wayland in the pipeline.

If I can replace the software renderer with Venus that would be enough for what I need.

30

Sorry I wasn’t very clear yesterday due to a lack of sleep. But by default Spice works just fine in Incus, it’s mainly when you configure Virtio-GPU Venus (in my case at last, but someone else has gotten it working so I’m thinking I have installation fault somewhere) that Spice doesn’t work.

The steps someone else mentioned earlier in this thread should work out of the box no problem and you can verify this with vulkaninfo –summary and with glxinfo -B in the guest.

1 Like
31

If I wanted to try Gnome Boxes or Virt-Manager on the host do I need to uninstall Incus or simply disable the service?

32

Incus can still be running, and you can even let a VM in Incus running while launching a VM with Gnome Boxes or Virt-Manager.

1 Like
33

Ok, installed Gnome Boxes and Virt-Manager and have again removed them. They do provide an out of the box UI option for OpenGL acceleration but neither had an obvious UI for multi-display output.

Additionally neither of them adjusted to the host monitor resolution when expanded to full-screen. Remote-viewer (virt-viewer) does adjust and also has CLI commands to create multiple guest displays and map them to multiple monitors on the host. This capability is verified in the Spice documentation:

Remote-viewer opens from the Incus command `incus console <vm_name> - -type=vga with a default of one display and when in the GUI application there is no UI option to add more displays. Is it possible to pass options for multi-monitor mapping via Incus CLI?

The more I read about remote-viewer the more impressive this little application appears.

Apparently remote-viewer can use unix sockets in addition to network sockets which means a complete bypass of the network stack.

34

Can Spice configuration be done outside Incus CLI? How does Incus interact with QEMU if it does not use QEMU libvirt?

Debian Trixie has LTS kernel 6.12 which means it cannot support a Venus/Vulkan solution:

An outdated blog post on clollabora described in 2021 how to enable 3D acceleration of Vulkan applications in QEMU through the Venus experimental Vulkan driver for VirtIO-GPU with a local development environment. Following up on the outdated write up, this is how its done today.

QEMU with VirtIO GPU Vulcan support

Make sure you have the proper version installed on the host:

linux kernel >= 6.13 built with CONFIG_UDMABUF
working Vulkan and kvm setup
qemu >= 9.2.0
virglrenderer with enabled venus support
mesa >= 24.2.0

If upgrading the kernel to 6.13.x is an easy change then maybe this is the way to go to get Venus instead of basic GL acceleration.

35

Incus drives QEMU through the QMP API.

libvirt is just another project driving QEMU in the backend for its VM, Incus and libvirt work in a pretty similar way in that regard, though Incus uses more QEMU features out of the box and is generally more opinionated than libvirt.

1 Like
36

The ability to implement Vulcan/Venus 3D acceleration in the guest is only just now emerging, requiring at least a 6.13 kernel which even the latest Debian release does not have.

However, if Incus is essentially a QEMU VMM via standard QEMU APIs is there any reason why any QEMU VMM compliant 3D acceleration method could not be applicable through Incus? What would be required to bring GL and Vulcan acceleration and Spice guest-multi-screen to host-multi-monitor mapping capability into Incus’ CLI command set? vDRM Native Context seems to be the future of VM graphics acceleration but for now Venus would be good enough for most applications and OS display.

Incus could open up to users a very desirable featureset that is clearly of at least a significant benefit to many users. There seems to be quite a large amount of work being put into these emerging graphics acceleration technologies.

Running remote-viewer Spice through Unix sockets in addition to the default network stack:
KVM/Qemu VMs with a multi-screen Spice console – III – local access with remote-viewer via a Unix socket

37

Incus is designed to do everything over a REST API so you can always interact with a remote Incus server rather than just locally.

For that reason, we have no interest in enabling a feature which only works when run on the local system.

You can play with raw.qemu and raw.qemu.conf to try and get it working, but shouldn’t expect any of that to land in Incus proper until we have a solution that correctly works for remote clients too.

1 Like
38

Maybe I’m looking at this from the wrong angle. IronRDP and good RDP implementations like Microsoft RDP seem designed from the ground up to punt graphics and other artifacts over the network and have the client handle everything.

IronRDP does not, however, explicitly address, what should be central for such a sophisticated technology, the concept and implementation of client-side GPU acceleration — other than to say that their Native Client (as opposed to their Web Client) implements software rendering. Their WebAssembly solition for browser rendering also does not explicitly address GPU acceleration but being in-browser it probably uses all the normal GPU rendering a web page would use — maybe.

It may also be that RDP is inherently, by design, accelerated by a client’s GPU. Strangely, also, no mention of Wayland that I could see.

Cloudflare has implemented IronRDP within its own infrastructure as a tool available to customers:

RDP with client-side GPU acceleration might be just the solution I’m looking for that provides near bare-metal performance for non-dedicated 3D rendering applications. Dedicated applications being games and 3D modelling programs.

I believe Incus is programmed in Go Lang whereas IronRDP is Rust so not sure if it would be a good candidate for some kind of Incus integration. But a some kind of excellent RDP implementation might punt Incus into top spot for all manner of VDI productivity solutions.

Incus won’t need Virtio-gpu solutions.

39

It’s not intuitive, but make sure you have the Spice vdagent and Qemu GA installed in your guest VM. That should enable autoresizing.

40

Their implementation language has no real relevance in this case as you can simply execute applications from Go with the `os/exec` package: exec package - os/exec - Go Packages

The need for virtio-gpu entirely depends on use-case. AFAIK Microsoft’s RDP graphics extensions, RemoteFX, were Windows only and were deprecated and removed a few years ago due to security vulnerabilities - how IronRDP and Devolution are using it is something I’m unsure about especially in conjunction with: Add GFX ([MS-RDPEGFX]) support · Issue #427 · Devolutions/IronRDP · GitHub .

1 Like