VMs: virgl and/or mdev GPU acceleration

I am playing around with these features at home, but I am not really sure if they are working or how to verify?

I created a new Impish VM using:
lxc launch --vm images:ubuntu/impish/desktop impish-desktop --console=vga

The virgl GPU is definitely present in the VM:

ubuntu@impish-desktop:~$ lspci | grep VGA
04:00.0 VGA compatible controller: Red Hat, Inc. Virtio GPU (rev 01)

Checking the renderer using glxinfo shows:

ubuntu@impish-desktop:~$ glxinfo | grep 'OpenGL renderer'
OpenGL renderer string: llvmpipe (LLVM 12.0.1, 256 bits)

If I am not mistaken llvmpipe means software right? Isn’t it supposed to say ‘virgl’ if the Virtio GPU is being used?

I decided to also try adding a mediated (mdev) GPU to the VM instead. After getting all BIOS, kernel modules and module parameters set correctly I can see the mdev types on the host:

anderson@anderson-nzxt:/sys/class/mdev_bus/0000:00:02.0/mdev_supported_types$ ls
i915-GVTg_V5_4  i915-GVTg_V5_8

They are showing as resources in LXD also:

anderson@anderson-nzxt:~$ lxc info --resources
GPUs:
  Card 0:
    NUMA node: 0
    Vendor: Intel Corporation (8086)
    Product: UHD Graphics 630 (Desktop) (3e92)
    PCI address: 0000:00:02.0
    Driver: i915 (5.4.0-89-generic)
    DRM:
      ID: 0
      Card: card0 (226:0)
      Control: controlD64 (226:0)
      Render: renderD128 (226:128)
    Mdev profiles:
      - i915-GVTg_V5_4 (1 available)
          low_gm_size: 128MB
          high_gm_size: 512MB
          fence: 4
          resolution: 1920x1200
          weight: 4
      - i915-GVTg_V5_8 (2 available)
          low_gm_size: 64MB
          high_gm_size: 384MB
          fence: 4
          resolution: 1024x768
          weight: 2

Adding the mdev GPU works:

anderson@anderson-nzxt:~$ lxc config device add impish-desktop i915 gpu gputype=mdev mdev=i915-GVTg_V5_8
Device i915 added to impish-desktop

However, I get the following error when I try to start the VM again:

anderson@anderson-nzxt:~$ lxc start --console=vga impish-desktop 
Error: Failed to start device "i915": VMs cannot match multiple GPUs per device
Try `lxc info --show-log impish-desktop` for more info

Not sure what that means other than the hunch that virgl and mdev are mutually exclusive when they both want to use the same GPU?

I am trying to understand how virgl and/or mdev GPUs work and so far don’t get the impression that I have GPU acceleration correctly enabled in the Impish VM?

Any help clarifying what is happening here would be much appreciated!

Looking at the code, this error suggests that multiple physical mdev devices are matching your selection criteria:

So try narrowing it down using the gpu LXD device settings:

https://linuxcontainers.org/lxd/docs/master/instances#gpu-mdev

And

lxc info --resources

Ah actually I just saw you had run lxc info --resources already, and that i915-GVTg_V5_8 profile has “2 available”.

@stgraber @monstermunchkin what do you think about this? Does it look like a bug? The card seems to have multiple mdev profiles, with multiple devices available on a single profile.

@tomp that’s not a bug. I, too, have multiple profiles with multiple devices. However, I don’t get the error.

Do you get the error if you specify the 2nd profile (as in this case)?

No, I don’t. Looking at the code, it seems, we’re missing a break statement after the vGPU has been created. That might be the issue here.

The output of lxc info --resources from @amcduffee suggests that there are multiple GPUs (due to Card 0). I only have this:

GPU:
  NUMA node: 0
  Vendor: Intel Corporation (8086)
  Product: UHD Graphics 620 (5917)
  PCI address: 0000:00:02.0
  Driver: i915 (5.13.0-20-generic)
  DRM:
    ID: 0
    Card: card0 (226:0)
    Control: controlD64 (226:0)
    Render: renderD128 (226:128)
  Mdev profiles:
    - i915-GVTg_V5_4 (1 available)
        low_gm_size: 128MB
        high_gm_size: 512MB
        fence: 4
        resolution: 1920x1200
        weight: 4
    - i915-GVTg_V5_8 (2 available)
        low_gm_size: 64MB
        high_gm_size: 384MB
        fence: 4
        resolution: 1024x768
        weight: 2

Ah, OK so @amcduffee try adding the id=0 config setting to specify which card:

lxc config device add impish-desktop i915 gpu gputype=mdev mdev=i915-GVTg_V5_8 id=0
1 Like

Adding id=0 worked. The VM boots and I can see the UHD graphics in addition to the Virtio GPU:

ubuntu@impish-desktop:~$ lspci | grep -e VGA -e UHD
04:00.0 VGA compatible controller: Red Hat, Inc. Virtio GPU (rev 01)
06:00.0 Display controller: Intel Corporation CometLake-S GT2 [UHD Graphics 630]

One other thing I noticed is that when the VM fails to start due to an mdev error the created mdev device isn’t cleaned up which leads to exhausting them on the host:

anderson@anderson-nzxt:~$ ls -la /sys/class/mdev_bus/0000\:00\:02.0/mdev_supported_types/i915-GVTg_V5_8/devices/
total 0
drwxr-xr-x 2 root root 0 Nov  8 08:31 .
drwxr-xr-x 3 root root 0 Nov  8 08:26 ..
anderson@anderson-nzxt:~$ lxc config device add impish-desktop i915 gpu gputype=mdev mdev=i915-GVTg_V5_8
Device i915 added to impish-desktop
anderson@anderson-nzxt:~$ lxc start --console=vga impish-desktop 
Error: Failed to start device "i915": VMs cannot match multiple GPUs per device
Try `lxc info --show-log impish-desktop` for more info
anderson@anderson-nzxt:~$ ls -la /sys/class/mdev_bus/0000\:00\:02.0/mdev_supported_types/i915-GVTg_V5_8/devices/
total 0
drwxr-xr-x 2 root root 0 Nov  8 08:31 .
drwxr-xr-x 3 root root 0 Nov  8 08:26 ..
lrwxrwxrwx 1 root root 0 Nov  8 08:56 57ced27c-89f3-448e-a288-0bff319cfd76 -> ../../../57ced27c-89f3-448e-a288-0bff319cfd76
anderson@anderson-nzxt:~$ lxc start --console=vga impish-desktop 
Error: Failed to start device "i915": VMs cannot match multiple GPUs per device
Try `lxc info --show-log impish-desktop` for more info
anderson@anderson-nzxt:~$ ls -la /sys/class/mdev_bus/0000\:00\:02.0/mdev_supported_types/i915-GVTg_V5_8/devices/
total 0
drwxr-xr-x 2 root root 0 Nov  8 08:31 .
drwxr-xr-x 3 root root 0 Nov  8 08:26 ..
lrwxrwxrwx 1 root root 0 Nov  8 08:56 57ced27c-89f3-448e-a288-0bff319cfd76 -> ../../../57ced27c-89f3-448e-a288-0bff319cfd76
lrwxrwxrwx 1 root root 0 Nov  8 08:56 f9874ac7-e41c-4b4f-b312-151c8d319175 -> ../../../f9874ac7-e41c-4b4f-b312-151c8d319175
anderson@anderson-nzxt:~$ lxc start --console=vga impish-desktop 
Error: Failed to start device "i915": No available mdev for profile "i915-GVTg_V5_8"
Try `lxc info --show-log impish-desktop` for more info

I can manually clean them up, but the error paths should be doing this right?:

anderson@anderson-nzxt:~$ echo 1 | sudo tee /sys/class/mdev_bus/0000\:00\:02.0/mdev_supported_types/i915-GVTg_V5_8/devices/*/remove
1
anderson@anderson-nzxt:~$ ls -la /sys/class/mdev_bus/0000\:00\:02.0/mdev_supported_types/i915-GVTg_V5_8/devices/
total 0
drwxr-xr-x 2 root root 0 Nov  8 08:31 .
drwxr-xr-x 3 root root 0 Nov  8 08:26 ..

Also, any idea on my first question regarding virgl not appearing to be enabled?

The general goal of this thread is to get GPU acceleration inside the VM. I don’t know that mdev is the right solution for my case because my primary monitor is 1440p and even the larger i915-GVTg_V5_4 profile only supports 1920x1200. I don’t know that virgl will work any better, or at all at 1440p, but I would like to at least give it a try.

For the clean up issue please could you open an issue over at https://github.com/lxc/lxd/issues so we can track the fix there.

Thanks

I’m afraid I’m not familiar with virgrl. I was of the understanding that passing a physical card through (mdev or otherwise) would allow acceleration of the output.

Have you tried passing through the whole card, or looked at whether your card supports SR-IOV GPU type?

Done.

1 Like

A bit more info from https://virgil3d.github.io:

Virgil is a research project to investigate the possibility of creating a virtual 3D GPU for use inside qemu virtual machines, that allows the guest operating system to use the capabilities of the host GPU to accelerate 3D rendering. The plan is to have a guest GPU that is fully independent of the host GPU.

The advantage it has over mdev and/or SR-IOV pass-through methods is it isn’t restricted to the number of physical GPUs or mdev profiles/devices on those GPUs. In my case, my discrete GPU is a Radeon RX 5700 XT which doesn’t support mediated (mdev) GPUs. I don’t want to pass the entire GPU through to a VM because I still want to use it on the host. Even if I was willing to pass the entire GPU through I can only do so to a single VM.

My goal here is to get GPU acceleration working inside multiple VMs simultaneously so that compositing effects and the desktop experience are more responsive.

On the QEMU command line virgl is typically enabled using the following and depends on the host having the virglrenderer library installed:

-device virtio-vga,virgl=on -display gtk,gl=on

I have taken a look at /var/snap/lxd/common/lxd/logs/impish-desktop/qemu.conf and don’t see the virgl option specified:

# GPU
[device "qemu_gpu"]
driver = "virtio-vga"
bus = "qemu_pcie3"
addr = "00.0"

I also checked the qemu-system-x86_64 command line, as launched by LXD, and don’t see these options there. So I figure based on these observations and glxinfo inside the VM showing llvmpipe as the OpenGL renderer that virgl acceleration isn’t setup.

Also, It looks like QEMU 6.1, which LXD is using, changed the naming scheme on the devices:
https://www.mail-archive.com/libvir-list@redhat.com/msg218308.html

So, going forward it may be necessary to have driver = "virtio-vga-gl" in the qemu.conf in order to use virgl.

Am I able to override these options using raw.qemu? If so, I will try to investigate this further. Although, I am not certain where the required virglrenderer library needs to be installed, in the snap or on the host?

If you can specify it on the qemu command line you should be able to use the raw.qemu option indeed.