VMs: virgl and/or mdev GPU acceleration

I am playing around with these features at home, but I am not really sure if they are working or how to verify?

I created a new Impish VM using:
lxc launch --vm images:ubuntu/impish/desktop impish-desktop --console=vga

The virgl GPU is definitely present in the VM:

ubuntu@impish-desktop:~$ lspci | grep VGA
04:00.0 VGA compatible controller: Red Hat, Inc. Virtio GPU (rev 01)

Checking the renderer using glxinfo shows:

ubuntu@impish-desktop:~$ glxinfo | grep 'OpenGL renderer'
OpenGL renderer string: llvmpipe (LLVM 12.0.1, 256 bits)

If I am not mistaken llvmpipe means software right? Isnā€™t it supposed to say ā€˜virglā€™ if the Virtio GPU is being used?

I decided to also try adding a mediated (mdev) GPU to the VM instead. After getting all BIOS, kernel modules and module parameters set correctly I can see the mdev types on the host:

anderson@anderson-nzxt:/sys/class/mdev_bus/0000:00:02.0/mdev_supported_types$ ls
i915-GVTg_V5_4  i915-GVTg_V5_8

They are showing as resources in LXD also:

anderson@anderson-nzxt:~$ lxc info --resources
GPUs:
  Card 0:
    NUMA node: 0
    Vendor: Intel Corporation (8086)
    Product: UHD Graphics 630 (Desktop) (3e92)
    PCI address: 0000:00:02.0
    Driver: i915 (5.4.0-89-generic)
    DRM:
      ID: 0
      Card: card0 (226:0)
      Control: controlD64 (226:0)
      Render: renderD128 (226:128)
    Mdev profiles:
      - i915-GVTg_V5_4 (1 available)
          low_gm_size: 128MB
          high_gm_size: 512MB
          fence: 4
          resolution: 1920x1200
          weight: 4
      - i915-GVTg_V5_8 (2 available)
          low_gm_size: 64MB
          high_gm_size: 384MB
          fence: 4
          resolution: 1024x768
          weight: 2

Adding the mdev GPU works:

anderson@anderson-nzxt:~$ lxc config device add impish-desktop i915 gpu gputype=mdev mdev=i915-GVTg_V5_8
Device i915 added to impish-desktop

However, I get the following error when I try to start the VM again:

anderson@anderson-nzxt:~$ lxc start --console=vga impish-desktop 
Error: Failed to start device "i915": VMs cannot match multiple GPUs per device
Try `lxc info --show-log impish-desktop` for more info

Not sure what that means other than the hunch that virgl and mdev are mutually exclusive when they both want to use the same GPU?

I am trying to understand how virgl and/or mdev GPUs work and so far donā€™t get the impression that I have GPU acceleration correctly enabled in the Impish VM?

Any help clarifying what is happening here would be much appreciated!

Looking at the code, this error suggests that multiple physical mdev devices are matching your selection criteria:

So try narrowing it down using the gpu LXD device settings:

https://linuxcontainers.org/lxd/docs/master/instances#gpu-mdev

And

lxc info --resources

Ah actually I just saw you had run lxc info --resources already, and that i915-GVTg_V5_8 profile has ā€œ2 availableā€.

@stgraber @monstermunchkin what do you think about this? Does it look like a bug? The card seems to have multiple mdev profiles, with multiple devices available on a single profile.

@tomp thatā€™s not a bug. I, too, have multiple profiles with multiple devices. However, I donā€™t get the error.

Do you get the error if you specify the 2nd profile (as in this case)?

No, I donā€™t. Looking at the code, it seems, weā€™re missing a break statement after the vGPU has been created. That might be the issue here.

The output of lxc info --resources from @amcduffee suggests that there are multiple GPUs (due to Card 0). I only have this:

GPU:
  NUMA node: 0
  Vendor: Intel Corporation (8086)
  Product: UHD Graphics 620 (5917)
  PCI address: 0000:00:02.0
  Driver: i915 (5.13.0-20-generic)
  DRM:
    ID: 0
    Card: card0 (226:0)
    Control: controlD64 (226:0)
    Render: renderD128 (226:128)
  Mdev profiles:
    - i915-GVTg_V5_4 (1 available)
        low_gm_size: 128MB
        high_gm_size: 512MB
        fence: 4
        resolution: 1920x1200
        weight: 4
    - i915-GVTg_V5_8 (2 available)
        low_gm_size: 64MB
        high_gm_size: 384MB
        fence: 4
        resolution: 1024x768
        weight: 2

Ah, OK so @amcduffee try adding the id=0 config setting to specify which card:

lxc config device add impish-desktop i915 gpu gputype=mdev mdev=i915-GVTg_V5_8 id=0
1 Like

Adding id=0 worked. The VM boots and I can see the UHD graphics in addition to the Virtio GPU:

ubuntu@impish-desktop:~$ lspci | grep -e VGA -e UHD
04:00.0 VGA compatible controller: Red Hat, Inc. Virtio GPU (rev 01)
06:00.0 Display controller: Intel Corporation CometLake-S GT2 [UHD Graphics 630]

One other thing I noticed is that when the VM fails to start due to an mdev error the created mdev device isnā€™t cleaned up which leads to exhausting them on the host:

anderson@anderson-nzxt:~$ ls -la /sys/class/mdev_bus/0000\:00\:02.0/mdev_supported_types/i915-GVTg_V5_8/devices/
total 0
drwxr-xr-x 2 root root 0 Nov  8 08:31 .
drwxr-xr-x 3 root root 0 Nov  8 08:26 ..
anderson@anderson-nzxt:~$ lxc config device add impish-desktop i915 gpu gputype=mdev mdev=i915-GVTg_V5_8
Device i915 added to impish-desktop
anderson@anderson-nzxt:~$ lxc start --console=vga impish-desktop 
Error: Failed to start device "i915": VMs cannot match multiple GPUs per device
Try `lxc info --show-log impish-desktop` for more info
anderson@anderson-nzxt:~$ ls -la /sys/class/mdev_bus/0000\:00\:02.0/mdev_supported_types/i915-GVTg_V5_8/devices/
total 0
drwxr-xr-x 2 root root 0 Nov  8 08:31 .
drwxr-xr-x 3 root root 0 Nov  8 08:26 ..
lrwxrwxrwx 1 root root 0 Nov  8 08:56 57ced27c-89f3-448e-a288-0bff319cfd76 -> ../../../57ced27c-89f3-448e-a288-0bff319cfd76
anderson@anderson-nzxt:~$ lxc start --console=vga impish-desktop 
Error: Failed to start device "i915": VMs cannot match multiple GPUs per device
Try `lxc info --show-log impish-desktop` for more info
anderson@anderson-nzxt:~$ ls -la /sys/class/mdev_bus/0000\:00\:02.0/mdev_supported_types/i915-GVTg_V5_8/devices/
total 0
drwxr-xr-x 2 root root 0 Nov  8 08:31 .
drwxr-xr-x 3 root root 0 Nov  8 08:26 ..
lrwxrwxrwx 1 root root 0 Nov  8 08:56 57ced27c-89f3-448e-a288-0bff319cfd76 -> ../../../57ced27c-89f3-448e-a288-0bff319cfd76
lrwxrwxrwx 1 root root 0 Nov  8 08:56 f9874ac7-e41c-4b4f-b312-151c8d319175 -> ../../../f9874ac7-e41c-4b4f-b312-151c8d319175
anderson@anderson-nzxt:~$ lxc start --console=vga impish-desktop 
Error: Failed to start device "i915": No available mdev for profile "i915-GVTg_V5_8"
Try `lxc info --show-log impish-desktop` for more info

I can manually clean them up, but the error paths should be doing this right?:

anderson@anderson-nzxt:~$ echo 1 | sudo tee /sys/class/mdev_bus/0000\:00\:02.0/mdev_supported_types/i915-GVTg_V5_8/devices/*/remove
1
anderson@anderson-nzxt:~$ ls -la /sys/class/mdev_bus/0000\:00\:02.0/mdev_supported_types/i915-GVTg_V5_8/devices/
total 0
drwxr-xr-x 2 root root 0 Nov  8 08:31 .
drwxr-xr-x 3 root root 0 Nov  8 08:26 ..

Also, any idea on my first question regarding virgl not appearing to be enabled?

The general goal of this thread is to get GPU acceleration inside the VM. I donā€™t know that mdev is the right solution for my case because my primary monitor is 1440p and even the larger i915-GVTg_V5_4 profile only supports 1920x1200. I donā€™t know that virgl will work any better, or at all at 1440p, but I would like to at least give it a try.

For the clean up issue please could you open an issue over at https://github.com/lxc/lxd/issues so we can track the fix there.

Thanks

Iā€™m afraid Iā€™m not familiar with virgrl. I was of the understanding that passing a physical card through (mdev or otherwise) would allow acceleration of the output.

Have you tried passing through the whole card, or looked at whether your card supports SR-IOV GPU type?

Done.

1 Like

A bit more info from https://virgil3d.github.io:

Virgil is a research project to investigate the possibility of creating a virtual 3D GPU for use inside qemu virtual machines, that allows the guest operating system to use the capabilities of the host GPU to accelerate 3D rendering. The plan is to have a guest GPU that is fully independent of the host GPU.

The advantage it has over mdev and/or SR-IOV pass-through methods is it isnā€™t restricted to the number of physical GPUs or mdev profiles/devices on those GPUs. In my case, my discrete GPU is a Radeon RX 5700 XT which doesnā€™t support mediated (mdev) GPUs. I donā€™t want to pass the entire GPU through to a VM because I still want to use it on the host. Even if I was willing to pass the entire GPU through I can only do so to a single VM.

My goal here is to get GPU acceleration working inside multiple VMs simultaneously so that compositing effects and the desktop experience are more responsive.

On the QEMU command line virgl is typically enabled using the following and depends on the host having the virglrenderer library installed:

-device virtio-vga,virgl=on -display gtk,gl=on

I have taken a look at /var/snap/lxd/common/lxd/logs/impish-desktop/qemu.conf and donā€™t see the virgl option specified:

# GPU
[device "qemu_gpu"]
driver = "virtio-vga"
bus = "qemu_pcie3"
addr = "00.0"

I also checked the qemu-system-x86_64 command line, as launched by LXD, and donā€™t see these options there. So I figure based on these observations and glxinfo inside the VM showing llvmpipe as the OpenGL renderer that virgl acceleration isnā€™t setup.

Also, It looks like QEMU 6.1, which LXD is using, changed the naming scheme on the devices:
https://www.mail-archive.com/libvir-list@redhat.com/msg218308.html

So, going forward it may be necessary to have driver = "virtio-vga-gl" in the qemu.conf in order to use virgl.

Am I able to override these options using raw.qemu? If so, I will try to investigate this further. Although, I am not certain where the required virglrenderer library needs to be installed, in the snap or on the host?

If you can specify it on the qemu command line you should be able to use the raw.qemu option indeed.

Were you able to get this working? Iā€™m also struggling to get virgil support in my VM, and Iā€™m not sure where I am meant to put any raw.qemu options for lxdā€¦

I never followed through with trying to change the GPU device driver in QEMU because at the time I wrote this post the raw.qemu option was the only thing available and it is a big hammer approach because it overrides the entire qemu.conf. Since then the ability to override individual sections or individual keys in a section was added:
https://linuxcontainers.org/lxd/docs/master/reference/instance_options/#instance-options-qemu

So, based on the above reference, it should be possible to override just the GPU driver with something similar to this under the config block of an LXD instance:

config:
  ...
  raw.qemu.conf: |-
    [device "qemu_gpu"]
    driver = "virtio-vga-gl"

However, it looks like the QEMU packaged with LXD doesnā€™t compile support for virgl:

anderson@anderson-ryzen9:~# ldd /snap/lxd/current/bin/qemu-system-x86_64 | grep virglrenderer
anderson@anderson-ryzen9:~#

So, as expected, QEMU complains with the following error and the instance fails to start:

qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/test-virgl/qemu.conf:243: 'virtio-vga-gl' is not a valid device model name

I believe this error is due to the fact that the QEMU packaged with LXD hasnā€™t enabled virglrenderer at build time. It is also possible that virtio-vga-gl isnā€™t the right device name anymore, but this could only be confirmed by using a QEMU that is known to have virglrenderer support enabled.

Even if virglrenderer support is compiled into the QEMU binary that comes with LXD I am not sure if it will be used when connecting to the VM console using lxc console because I believe that LXD is just setting up a spice server and then forwarding the spice server socket.

It still isnā€™t clear to me if SPICE consoles will benefit from virgl being on in the guest. It is presumably possible to get the SPICE console accelerated with GL, but I think this is the SPICE server directly using GL calls on the host to speed up how it handles and processes the frame buffer images it captures, from the guest, and sends to the client. I donā€™t think it is related to whether or not the guest is using virgl, but I am unsure until I can research it further.

Right, the reason why we donā€™t build or try to enable VirGL is that from my understanding it only really works with SPICE when run on the same system that runs the VM. Thereā€™s never any such guarantee with LXD as everything, even local requests always go over our REST API.

Another issue with VirGL is the need for a full stack of DRI/DRM drivers be present and loadable by qemu. Not only does this potentially expand the attack surface of qemu quite significantly (because weā€™d need to allow for a lot of extra libs and files to be loaded and accessed) but it also is a massive pain to do from a snap or flatpak package where the application (LXD) is purposefully isolated from the host system.

So weā€™d either need to jump through a lot of hoops to get to load those libraries from the host system, at risk of massive crashes depending on versions and distributions used, or weā€™d need to bundle them all along with LXD which would be pretty painful for Intel/AMD and downright not possible for NVIDIA (due to licensing and libs having to line up with the driver).

1 Like