I’m trying to run OMV inside an incus VM and tried to do a disk passthrough using the disk device. Inside the VM, the disk is emulated and I can’t access SMART functionality, for example.
Is there a proper way to pass the whole disk in like it can be done with NIC devices?
If the disk is NVMe or is alone in its SCSI controller, then you can pass the full PCI device and get the same behavior as a NIC.
But if it’s on a shared controller, then no, it will be connected to the host kernel and then accessed from there.
It’s in a SATA controller, but all such disks need to move. So I’ll just try to pass that through.
Thanks a lot for the fast response.
Couldn’t do it.
The relevant PCI device I think is the following:
victoitor@deu-bereu:~$ incus info --resources
(...)
Device 19:
Address: 0000:01:00.1
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: 500 Series Chipset SATA Controller
Product ID: 43eb
NUMA node: 0
IOMMU group: 8
Driver: ahci
This is the profile I’m running:
victoitor@deu-bereu:~$ incus profile show openmediavault
config:
cloud-init.vendor-data: |-
#cloud-config
timezone: America/Fortaleza
package_update: true
package_upgrade: true
package_reboot_if_required: true
packages:
- wget
limits.memory: 8GiB
description: OpenMediaVault VM profile
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: lvm
size: 10GiB
type: disk
sata:
address: "01:00.1"
type: pci
name: openmediavault
used_by:
- /1.0/instances/omv-power
project: default
The situation is that the instance doesn’t start with the error being qemu-system-x86_64:/run/incus/omv-power/qemu.conf:271: vfio 0000:01:00.1: group 8 is not viable Please ensure all devices within the iommu_group are bound to their vfio bus driver.
Not sure it helps, but this is the full log of instance start:
victoitor@deu-bereu:~$ incus start omv-power
Error: Failed to run: forklimits limit=memlock:unlimited:unlimited fd=3 fd=4 -- /opt/incus/bin/qemu-system-x86_64 -S -name omv-power -uuid 293f1a2f-7b19-43a1-81b8-857ea24c1446 -daemonize -cpu host,hv_passthrough -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=allow,resourcecontrol=deny -readconfig /run/incus/omv-power/qemu.conf -spice unix=on,disable-ticketing=on,addr=/run/incus/omv-power/qemu.spice -pidfile /run/incus/omv-power/qemu.pid -D /var/log/incus/omv-power/qemu.log -smbios type=2,manufacturer=LinuxContainers,product=Incus -runas incus: : exit status 1
Try `incus info --show-log omv-power` for more info
victoitor@deu-bereu:~$ incus info --show-log omv-power
Name: omv-power
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: 2024/07/21 18:50 -03
Last Used: 2024/07/21 22:52 -03
Snapshots:
+-------------+----------------------+------------+----------+
| NAME | TAKEN AT | EXPIRES AT | STATEFUL |
+-------------+----------------------+------------+----------+
| incus-agent | 2024/07/21 22:56 -03 | | NO |
+-------------+----------------------+------------+----------+
Log:
qemu-system-x86_64:/run/incus/omv-power/qemu.conf:271: vfio 0000:01:00.1: group 8 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.
This is the full resources on the machine I have.
victoitor@deu-bereu:~$ incus info --resources
System:
UUID: 03560274-043c-05db-8506-0c0700080009
Vendor: Gigabyte Technology Co., Ltd.
Product: A520M DS3H V2
Family: A520 MB
Version: -CF
SKU: Default string
Serial: Default string
Type: physical
Chassis:
Vendor: Default string
Type: Desktop
Version: Default string
Serial: Default string
Motherboard:
Vendor: Gigabyte Technology Co., Ltd.
Product: A520M DS3H V2
Serial: Default string
Version: x.x
Firmware:
Vendor: American Megatrends International, LLC.
Version: FB
Date: 06/08/2023
Load:
Processes: 247
Average: 0.08 0.02 0.01
CPU:
Architecture: x86_64
Vendor: AuthenticAMD
Name: AMD Ryzen 5 5600GT with Radeon Graphics
Caches:
- Level 1 (type: Data): 32KiB
- Level 1 (type: Instruction): 32KiB
- Level 2 (type: Unified): 512KiB
- Level 3 (type: Unified): 16MiB
Cores:
- Core 0
Frequency: 1400Mhz
Threads:
- 0 (id: 0, online: true, NUMA node: 0)
- 1 (id: 6, online: true, NUMA node: 0)
- Core 1
Frequency: 1395Mhz
Threads:
- 0 (id: 1, online: true, NUMA node: 0)
- 1 (id: 7, online: true, NUMA node: 0)
- Core 2
Frequency: 1396Mhz
Threads:
- 0 (id: 2, online: true, NUMA node: 0)
- 1 (id: 8, online: true, NUMA node: 0)
- Core 3
Frequency: 1400Mhz
Threads:
- 0 (id: 3, online: true, NUMA node: 0)
- 1 (id: 9, online: true, NUMA node: 0)
- Core 4
Frequency: 1400Mhz
Threads:
- 0 (id: 10, online: true, NUMA node: 0)
- 1 (id: 4, online: true, NUMA node: 0)
- Core 5
Frequency: 1400Mhz
Threads:
- 0 (id: 11, online: true, NUMA node: 0)
- 1 (id: 5, online: true, NUMA node: 0)
Frequency: 1398Mhz (min: 1400Mhz, max: 4668Mhz)
Memory:
Free: 61.34GiB
Used: 4.66GiB
Total: 66.00GiB
GPU:
NUMA node: 0
Vendor: Advanced Micro Devices, Inc. [AMD/ATI] (1002)
Product: Cezanne [Radeon Vega Series / Radeon Vega Mobile Series] (1638)
PCI address: 0000:05:00.0
Driver: amdgpu (6.1.0-23-amd64)
DRM:
ID: 0
Card: card0 (226:0)
Control: controlD64 (226:0)
Render: renderD128 (226:128)
NIC:
NUMA node: 0
Vendor: Realtek Semiconductor Co., Ltd. (10ec)
Product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (8168)
PCI address: 0000:03:00.0
Driver: r8169 (6.1.0-23-amd64)
Ports:
- Port 0 (ethernet)
ID: enp3s0
Address: 74:56:3c:db:85:0c
Supported modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full
Supported ports: twisted pair, media-independent
Port type: twisted pair
Transceiver type: external
Auto negotiation: true
Link detected: true
Link speed: 1000Mbit/s (full duplex)
Disks:
Disk 0:
NUMA node: 0
ID: nvme0n1
Device: 259:0
Model: WD Blue SN570 250GB
Type: nvme
Size: 232.89GiB
WWN: eui.e8238fa6bf530001001b448b4e6dde69
Read-Only: false
Removable: false
Partitions:
- Partition 1
ID: nvme0n1p1
Device: 259:1
Read-Only: false
Size: 953.00MiB
- Partition 2
ID: nvme0n1p2
Device: 259:2
Read-Only: false
Size: 231.95GiB
Disk 1:
NUMA node: 0
ID: sda
Device: 8:0
Model: ST4000VN006-3CW104
Type: sata
Size: 3.64TiB
Read-Only: false
Removable: false
Disk 2:
NUMA node: 0
ID: sdb
Device: 8:16
Model: ST4000VN006-3CW104
Type: sata
Size: 3.64TiB
Read-Only: false
Removable: false
Disk 3:
NUMA node: 0
ID: sdc
Device: 8:32
Model: ST4000VN006-3CW104
Type: sata
Size: 3.64TiB
Read-Only: false
Removable: false
PCI devices:
Device 0:
Address: 0000:00:00.0
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir/Cezanne Root Complex
Product ID: 1630
NUMA node: 0
IOMMU group: 0
Driver:
Device 1:
Address: 0000:00:00.2
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir/Cezanne IOMMU
Product ID: 1631
NUMA node: 0
IOMMU group: 0
Driver:
Device 2:
Address: 0000:00:01.0
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir PCIe Dummy Host Bridge
Product ID: 1632
NUMA node: 0
IOMMU group: 0
Driver:
Device 3:
Address: 0000:00:02.0
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir PCIe Dummy Host Bridge
Product ID: 1632
NUMA node: 0
IOMMU group: 1
Driver:
Device 4:
Address: 0000:00:02.1
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir/Cezanne PCIe GPP Bridge
Product ID: 1634
NUMA node: 0
IOMMU group: 2
Driver: pcieport
Device 5:
Address: 0000:00:02.2
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir/Cezanne PCIe GPP Bridge
Product ID: 1634
NUMA node: 0
IOMMU group: 3
Driver: pcieport
Device 6:
Address: 0000:00:08.0
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir PCIe Dummy Host Bridge
Product ID: 1632
NUMA node: 0
IOMMU group: 4
Driver:
Device 7:
Address: 0000:00:08.1
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir Internal PCIe GPP Bridge to Bus
Product ID: 1635
NUMA node: 0
IOMMU group: 5
Driver: pcieport
Device 8:
Address: 0000:00:14.0
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: FCH SMBus Controller
Product ID: 790b
NUMA node: 0
IOMMU group: 6
Driver:
Device 9:
Address: 0000:00:14.3
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: FCH LPC Bridge
Product ID: 790e
NUMA node: 0
IOMMU group: 6
Driver:
Device 10:
Address: 0000:00:18.0
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Cezanne Data Fabric; Function 0
Product ID: 166a
NUMA node: 0
IOMMU group: 7
Driver:
Device 11:
Address: 0000:00:18.1
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Cezanne Data Fabric; Function 1
Product ID: 166b
NUMA node: 0
IOMMU group: 7
Driver:
Device 12:
Address: 0000:00:18.2
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Cezanne Data Fabric; Function 2
Product ID: 166c
NUMA node: 0
IOMMU group: 7
Driver:
Device 13:
Address: 0000:00:18.3
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Cezanne Data Fabric; Function 3
Product ID: 166d
NUMA node: 0
IOMMU group: 7
Driver: k10temp
Device 14:
Address: 0000:00:18.4
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Cezanne Data Fabric; Function 4
Product ID: 166e
NUMA node: 0
IOMMU group: 7
Driver:
Device 15:
Address: 0000:00:18.5
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Cezanne Data Fabric; Function 5
Product ID: 166f
NUMA node: 0
IOMMU group: 7
Driver:
Device 16:
Address: 0000:00:18.6
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Cezanne Data Fabric; Function 6
Product ID: 1670
NUMA node: 0
IOMMU group: 7
Driver:
Device 17:
Address: 0000:00:18.7
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Cezanne Data Fabric; Function 7
Product ID: 1671
NUMA node: 0
IOMMU group: 7
Driver:
Device 18:
Address: 0000:01:00.0
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product:
Product ID: 43ec
NUMA node: 0
IOMMU group: 8
Driver: xhci_hcd
Device 19:
Address: 0000:01:00.1
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: 500 Series Chipset SATA Controller
Product ID: 43eb
NUMA node: 0
IOMMU group: 8
Driver: ahci
Device 20:
Address: 0000:01:00.2
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: 500 Series Chipset Switch Upstream Port
Product ID: 43e9
NUMA node: 0
IOMMU group: 8
Driver: pcieport
Device 21:
Address: 0000:02:03.0
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product:
Product ID: 43ea
NUMA node: 0
IOMMU group: 8
Driver: pcieport
Device 22:
Address: 0000:03:00.0
Vendor: Realtek Semiconductor Co., Ltd.
Vendor ID: 10ec
Product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
Product ID: 8168
NUMA node: 0
IOMMU group: 8
Driver: r8169
Device 23:
Address: 0000:04:00.0
Vendor: Sandisk Corp
Vendor ID: 15b7
Product: WD Blue SN570 NVMe SSD 1TB
Product ID: 501a
NUMA node: 0
IOMMU group: 9
Driver: nvme
Device 24:
Address: 0000:05:00.0
Vendor: Advanced Micro Devices, Inc. [AMD/ATI]
Vendor ID: 1002
Product: Cezanne [Radeon Vega Series / Radeon Vega Mobile Series]
Product ID: 1638
NUMA node: 0
IOMMU group: 10
Driver: amdgpu
Device 25:
Address: 0000:05:00.1
Vendor: Advanced Micro Devices, Inc. [AMD/ATI]
Vendor ID: 1002
Product: Renoir Radeon High Definition Audio Controller
Product ID: 1637
NUMA node: 0
IOMMU group: 11
Driver: snd_hda_intel
Device 26:
Address: 0000:05:00.2
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Family 17h (Models 10h-1fh) Platform Security Processor
Product ID: 15df
NUMA node: 0
IOMMU group: 12
Driver: ccp
Device 27:
Address: 0000:05:00.3
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir/Cezanne USB 3.1
Product ID: 1639
NUMA node: 0
IOMMU group: 13
Driver: xhci_hcd
Device 28:
Address: 0000:05:00.4
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir/Cezanne USB 3.1
Product ID: 1639
NUMA node: 0
IOMMU group: 14
Driver: xhci_hcd
Device 29:
Address: 0000:05:00.6
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Family 17h/19h HD Audio Controller
Product ID: 15e3
NUMA node: 0
IOMMU group: 15
Driver: snd_hda_intel
There is a bunch of stuff in IOMMU group 8. Basically devices 18 through 22. Is it either all or nothing?
Yeah, unfortunately that’s how PCI pasthrough works at the hardware level, everything sharing the same group must be moved together.
If anyone came here looking for an alternative solution, I will just add that it might be possible to do this by adding a pci sata controller card and passing that card through to the vm. Found this solution online, but haven’t tried it yet.