PCI/GPU passthrough for VMs - 3.23+

Apologies if this question has already been answered …

I have a use case where I need to manage containers and VMs on a cluster in an integrated fashion. Some of the VMs need access to a raw NVIDIA GPU device (without there being any drivers on the LXD server).

I see that 3.20+ brings the ability to configure a physical-nic device (using QEMU PCI passthrough) for a VM, and was wondering if similar capability is available or forthcoming for a generic (non-NIC) device.

Thanks,
-Vijay

Not currently but it’s on our short term roadmap.
Until then, it’s possible to directly do it by passing the right arguments directly to qemu using the raw.qemu config key. We had a few users do that successfully.

1 Like

Thanks for the quick reply!
Is there a pointer you can share for documentation on how to use the raw.qemu key … will also search these forums on my own.

Thanks,
-Vijay

@morphis can you share what you’ve been doing?

Never mind, figured this out …

In case someone else is interested, describing what I did below:

After doing the usual IOMMU and vfio stuff (to isolate the desired PCI device from the host), the device can be passed through to the host as shown in the commands below:

akriadmin@c4akri01:~$ lxc profile show vm
config:
  raw.qemu: -device vfio-pci,host=41:00.0
description: LXD profile for virtual machines
devices: {}
name: vm
used_by: []
akriadmin@c4akri01:~$ lxc launch --target c4akri01 --vm images:ubuntu/18.04 --profile default --profile vm
Creating the instance
Instance name is: loving-peacock
Starting loving-peacock
akriadmin@c4akri01:~$
1 Like

@Vijay_Karamcheti Yes, PCI passthrough via vfio is what I did as well.

If your card is in use by another driver (for example in case of a GPU) you have to unbind it first. For GPUs add the following to /etc/default/grub (NOTE: The ID is the <vendor>:<product> combination you can find with lspci -nn)

GRUB_CMDLINE_LINUX_DEFAULT="video=vesafb:off,efifb:off vfio-pci.ids=1002:67c7,1002:aaf0"

Where the PCI ids are the ones of your GPU. In this case the GPU has also an audio PCI device which has to be passed to VFIO within the same group, otherwise Qemu will fail to forward. Also add

vfio-pci 
vfio_iommu_type1

to your /etc/modules and create /etc/modprobe.d/vfio.conf with the following content

options vfio-pci ids=1002:67c7,1002:aaf0
softdep radeon pre: vfio-pci
softdep amdgpu pre: vfio-pci
softdep nouveau pre: vfio-pci
softdep drm pre: vfio-pci

Now run

$ update-grub
$ update-initramfs -u -k all

and finally reboot the system. Once back up the PCI device can now be forwarded via a raw.qemu: -device vfio-pci,host=01:00.0.

Hope this helps!

1 Like