Physical GPU passthrough to VM seems to be working, but guests output only to Virtio GPU

Hello, I’m fairly n00b with LXD, coming from some other virtualization/hypervisor systems (VMWare, Proxmox, virt-manager on this same Ubuntu machine that I am running LXD on).

My use case is to run VMs with a GPU and USB controller passed through, with a physical monitor, keyboard, and mouse.

A recurring problem for me is that even though I seem to have successfully passed through a GPU successfully (e.g., on the command line inside the VM, it shows up when using the lspci command[1]), the system only outputs video to the virtual GPU.

In this example I am using Pop OS, but I have had the same issue with several distros. (I chose Pop OS today because I know after installation it defaults to using all GPUs/monitors that are available.)

I created a VM named pop using a local ISO image (this OS has NVIDIA drivers pre-installed). I ran the installer, rebooted, and confirmed it all works with the virtual VGA console with lxc console pop --type vga. I then stopped it and added my GPU and USB controller card with:

lxc config device add pop rtx3060 gpu gputype=physical pci=0000:01:00.0
Device rtx3060 added to pop
lxc config device add pop usbcard pci address=0000:06:00.0
Device usbcard added to pop

The USB card works fine. I plug in mice, and keyboards, and they work.

The GPU almost works: running nvidia-smi sees the GPU (an RTX 3060), and the NVIDIA Settings GUI application can not only see the GPU, but it shows that an “LG Electronics LG TV” is connected to it, which is indeed the test display I have connected.

But… the display is showing “no signal”.

I’ve tried using different cables and monitors, with the same results.

So it looks so close and yet, not working at all. It would seem like it could be a PopOS problem, except I have had the same issue with Manjaro and all the other distros I have tried, so I think the problem must be some kind of user error that is eluding me…

Any pointers or ideas of things to try would be appreciated.

Thanks!


[1] Example of how in the VM I can see the physical GPU, even though the OS in the VM is only displaying on the Virtio GPU.

mason@pop-os:~$ lspci -k | grep -A 2 -E "VGA"
04:00.0 VGA compatible controller: Red Hat, Inc. Virtio GPU (rev 01)
	Subsystem: Red Hat, Inc. Virtio GPU
	Kernel driver in use: virtio-pci
--
07:00.0 VGA compatible controller: NVIDIA Corporation GA106 [GeForce RTX 3060] (rev a1)
	Subsystem: Gigabyte Technology Co., Ltd GA106 [GeForce RTX 3060]
	Kernel driver in use: nvidia
mason@pop-os:~$ 

## Or the same thing on Arch (a VM in this case created from the LXD image server)

[root@archlinux ~]# lspci -k | grep -A 2 -E "(VGA|3D)"
04:00.0 VGA compatible controller: Red Hat, Inc. Virtio GPU (rev 01)
	Subsystem: Red Hat, Inc. Device 1100
	Kernel driver in use: virtio-pci
--
07:00.0 VGA compatible controller: NVIDIA Corporation GA106 [GeForce RTX 3060] (rev a1)
	Subsystem: Gigabyte Technology Co., Ltd Device 4074
	Kernel driver in use: nouveau
[root@archlinux ~]#
1 Like

Oh, one thing I should add is that I did get these same GPUs working with Windows (with all the same hardware, displays, etc), by following the “Windows 11 VM on LXD” video on the LXD YouTube channel. (In fact, that video was the thing that got me to try LXD in the first place.)

So whatever the problem is, it seems specific to Linux guests.

I had a similar problem, configuring xorg manually everything was solved.

EX:

Section “Device”
Identifier “Device0”
Driver “nvidia”
BusID “4:0:0”
EndSection

2 Likes

Thanks! That worked.

I’d tried a more elaborate xorg.conf before, but something must have been wrong with it. Using something very close to your suggestion made both GPUs and monitors start working with this VM.

The exact contents of my now-working xorg.conf are shown below for posterity. I note that this does disable the virtual VGA device, so lxc console pop --type vga no longer works. I could probably tinker around and figure out how add that virtual device to this xorg.conf, but I don’t need that for this use case, so I didn’t investigate further.

Thanks again!

(I do wonder what prevents PopOS and these other distros from detecting the attached monitors automatically, like they do when freshly installed onto physical hardware, but I don’t need to know the answer to that…)

 mason@pop-os:~$ cat /usr/share/X11/xorg.conf.d/xorg.conf

Section "Device"
	Identifier  "RTX3060"
	Driver      "nvidia"
	BusID       "PCI:7:0:0"
EndSection

Section "Device"
	Identifier  "GTX1660"
	Driver      "nvidia"
	BusID       "PCI:8:0:0"
EndSection

P.S.
Just for the interest of potential future searchers, I will also append the config of the VM in which this is working:

➜  ~ lxc config show pop
architecture: x86_64
config:
  image.architecture: x86_64
  image.description: Pop OS test (20221014)
  image.os: poplord
  image.release: "22.04"
  limits.cpu: "12"
  limits.memory: 24GB
  security.secureboot: "false"
  volatile.base_image: fd08795759096a49cf44acc81a9917e59ba084458325083707460678455c81fd
  volatile.cloud-init.instance-id: 378c5452-beef-beef-b29e-5d766b445bc7
  volatile.eth0.host_name: tap92884c55
  volatile.eth0.hwaddr: re:da:ct:ed:du:de
  volatile.gtx1060.last_state.pci.driver: vfio-pci
  volatile.gtx1060.last_state.pci.slot.name: "0000:07:00.0"
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.rtx3060a.last_state.pci.driver: vfio-pci
  volatile.rtx3060a.last_state.pci.slot.name: "0000:01:00.0"
  volatile.usbcard.last_state.pci.driver: vfio-pci
  volatile.usbcard.last_state.pci.slot.name: "0000:06:00.0"
  volatile.uuid: f2018289-6e5f-beef-beef-c8d6cfc3d760
  volatile.vsock_id: "13"
devices:
  gtx1060:
    gputype: physical
    pci: "0000:07:00.0"
    type: gpu
  rtx3060a:
    gputype: physical
    pci: "0000:01:00.0"
    type: gpu
  usbcard:
    address: "0000:06:00.0"
    type: pci
ephemeral: false
profiles:
- default
- bridgeprofile
stateful: false
description: ""
➜  ~