Couldn’t do it.
The relevant PCI device I think is the following:
victoitor@deu-bereu:~$ incus info --resources
(...)
Device 19:
Address: 0000:01:00.1
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: 500 Series Chipset SATA Controller
Product ID: 43eb
NUMA node: 0
IOMMU group: 8
Driver: ahci
This is the profile I’m running:
victoitor@deu-bereu:~$ incus profile show openmediavault
config:
cloud-init.vendor-data: |-
#cloud-config
timezone: America/Fortaleza
package_update: true
package_upgrade: true
package_reboot_if_required: true
packages:
- wget
limits.memory: 8GiB
description: OpenMediaVault VM profile
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: lvm
size: 10GiB
type: disk
sata:
address: "01:00.1"
type: pci
name: openmediavault
used_by:
- /1.0/instances/omv-power
project: default
The situation is that the instance doesn’t start with the error being qemu-system-x86_64:/run/incus/omv-power/qemu.conf:271: vfio 0000:01:00.1: group 8 is not viable Please ensure all devices within the iommu_group are bound to their vfio bus driver.
Not sure it helps, but this is the full log of instance start:
victoitor@deu-bereu:~$ incus start omv-power
Error: Failed to run: forklimits limit=memlock:unlimited:unlimited fd=3 fd=4 -- /opt/incus/bin/qemu-system-x86_64 -S -name omv-power -uuid 293f1a2f-7b19-43a1-81b8-857ea24c1446 -daemonize -cpu host,hv_passthrough -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=allow,resourcecontrol=deny -readconfig /run/incus/omv-power/qemu.conf -spice unix=on,disable-ticketing=on,addr=/run/incus/omv-power/qemu.spice -pidfile /run/incus/omv-power/qemu.pid -D /var/log/incus/omv-power/qemu.log -smbios type=2,manufacturer=LinuxContainers,product=Incus -runas incus: : exit status 1
Try `incus info --show-log omv-power` for more info
victoitor@deu-bereu:~$ incus info --show-log omv-power
Name: omv-power
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: 2024/07/21 18:50 -03
Last Used: 2024/07/21 22:52 -03
Snapshots:
+-------------+----------------------+------------+----------+
| NAME | TAKEN AT | EXPIRES AT | STATEFUL |
+-------------+----------------------+------------+----------+
| incus-agent | 2024/07/21 22:56 -03 | | NO |
+-------------+----------------------+------------+----------+
Log:
qemu-system-x86_64:/run/incus/omv-power/qemu.conf:271: vfio 0000:01:00.1: group 8 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.
This is the full resources on the machine I have.
victoitor@deu-bereu:~$ incus info --resources
System:
UUID: 03560274-043c-05db-8506-0c0700080009
Vendor: Gigabyte Technology Co., Ltd.
Product: A520M DS3H V2
Family: A520 MB
Version: -CF
SKU: Default string
Serial: Default string
Type: physical
Chassis:
Vendor: Default string
Type: Desktop
Version: Default string
Serial: Default string
Motherboard:
Vendor: Gigabyte Technology Co., Ltd.
Product: A520M DS3H V2
Serial: Default string
Version: x.x
Firmware:
Vendor: American Megatrends International, LLC.
Version: FB
Date: 06/08/2023
Load:
Processes: 247
Average: 0.08 0.02 0.01
CPU:
Architecture: x86_64
Vendor: AuthenticAMD
Name: AMD Ryzen 5 5600GT with Radeon Graphics
Caches:
- Level 1 (type: Data): 32KiB
- Level 1 (type: Instruction): 32KiB
- Level 2 (type: Unified): 512KiB
- Level 3 (type: Unified): 16MiB
Cores:
- Core 0
Frequency: 1400Mhz
Threads:
- 0 (id: 0, online: true, NUMA node: 0)
- 1 (id: 6, online: true, NUMA node: 0)
- Core 1
Frequency: 1395Mhz
Threads:
- 0 (id: 1, online: true, NUMA node: 0)
- 1 (id: 7, online: true, NUMA node: 0)
- Core 2
Frequency: 1396Mhz
Threads:
- 0 (id: 2, online: true, NUMA node: 0)
- 1 (id: 8, online: true, NUMA node: 0)
- Core 3
Frequency: 1400Mhz
Threads:
- 0 (id: 3, online: true, NUMA node: 0)
- 1 (id: 9, online: true, NUMA node: 0)
- Core 4
Frequency: 1400Mhz
Threads:
- 0 (id: 10, online: true, NUMA node: 0)
- 1 (id: 4, online: true, NUMA node: 0)
- Core 5
Frequency: 1400Mhz
Threads:
- 0 (id: 11, online: true, NUMA node: 0)
- 1 (id: 5, online: true, NUMA node: 0)
Frequency: 1398Mhz (min: 1400Mhz, max: 4668Mhz)
Memory:
Free: 61.34GiB
Used: 4.66GiB
Total: 66.00GiB
GPU:
NUMA node: 0
Vendor: Advanced Micro Devices, Inc. [AMD/ATI] (1002)
Product: Cezanne [Radeon Vega Series / Radeon Vega Mobile Series] (1638)
PCI address: 0000:05:00.0
Driver: amdgpu (6.1.0-23-amd64)
DRM:
ID: 0
Card: card0 (226:0)
Control: controlD64 (226:0)
Render: renderD128 (226:128)
NIC:
NUMA node: 0
Vendor: Realtek Semiconductor Co., Ltd. (10ec)
Product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (8168)
PCI address: 0000:03:00.0
Driver: r8169 (6.1.0-23-amd64)
Ports:
- Port 0 (ethernet)
ID: enp3s0
Address: 74:56:3c:db:85:0c
Supported modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full
Supported ports: twisted pair, media-independent
Port type: twisted pair
Transceiver type: external
Auto negotiation: true
Link detected: true
Link speed: 1000Mbit/s (full duplex)
Disks:
Disk 0:
NUMA node: 0
ID: nvme0n1
Device: 259:0
Model: WD Blue SN570 250GB
Type: nvme
Size: 232.89GiB
WWN: eui.e8238fa6bf530001001b448b4e6dde69
Read-Only: false
Removable: false
Partitions:
- Partition 1
ID: nvme0n1p1
Device: 259:1
Read-Only: false
Size: 953.00MiB
- Partition 2
ID: nvme0n1p2
Device: 259:2
Read-Only: false
Size: 231.95GiB
Disk 1:
NUMA node: 0
ID: sda
Device: 8:0
Model: ST4000VN006-3CW104
Type: sata
Size: 3.64TiB
Read-Only: false
Removable: false
Disk 2:
NUMA node: 0
ID: sdb
Device: 8:16
Model: ST4000VN006-3CW104
Type: sata
Size: 3.64TiB
Read-Only: false
Removable: false
Disk 3:
NUMA node: 0
ID: sdc
Device: 8:32
Model: ST4000VN006-3CW104
Type: sata
Size: 3.64TiB
Read-Only: false
Removable: false
PCI devices:
Device 0:
Address: 0000:00:00.0
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir/Cezanne Root Complex
Product ID: 1630
NUMA node: 0
IOMMU group: 0
Driver:
Device 1:
Address: 0000:00:00.2
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir/Cezanne IOMMU
Product ID: 1631
NUMA node: 0
IOMMU group: 0
Driver:
Device 2:
Address: 0000:00:01.0
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir PCIe Dummy Host Bridge
Product ID: 1632
NUMA node: 0
IOMMU group: 0
Driver:
Device 3:
Address: 0000:00:02.0
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir PCIe Dummy Host Bridge
Product ID: 1632
NUMA node: 0
IOMMU group: 1
Driver:
Device 4:
Address: 0000:00:02.1
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir/Cezanne PCIe GPP Bridge
Product ID: 1634
NUMA node: 0
IOMMU group: 2
Driver: pcieport
Device 5:
Address: 0000:00:02.2
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir/Cezanne PCIe GPP Bridge
Product ID: 1634
NUMA node: 0
IOMMU group: 3
Driver: pcieport
Device 6:
Address: 0000:00:08.0
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir PCIe Dummy Host Bridge
Product ID: 1632
NUMA node: 0
IOMMU group: 4
Driver:
Device 7:
Address: 0000:00:08.1
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir Internal PCIe GPP Bridge to Bus
Product ID: 1635
NUMA node: 0
IOMMU group: 5
Driver: pcieport
Device 8:
Address: 0000:00:14.0
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: FCH SMBus Controller
Product ID: 790b
NUMA node: 0
IOMMU group: 6
Driver:
Device 9:
Address: 0000:00:14.3
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: FCH LPC Bridge
Product ID: 790e
NUMA node: 0
IOMMU group: 6
Driver:
Device 10:
Address: 0000:00:18.0
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Cezanne Data Fabric; Function 0
Product ID: 166a
NUMA node: 0
IOMMU group: 7
Driver:
Device 11:
Address: 0000:00:18.1
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Cezanne Data Fabric; Function 1
Product ID: 166b
NUMA node: 0
IOMMU group: 7
Driver:
Device 12:
Address: 0000:00:18.2
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Cezanne Data Fabric; Function 2
Product ID: 166c
NUMA node: 0
IOMMU group: 7
Driver:
Device 13:
Address: 0000:00:18.3
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Cezanne Data Fabric; Function 3
Product ID: 166d
NUMA node: 0
IOMMU group: 7
Driver: k10temp
Device 14:
Address: 0000:00:18.4
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Cezanne Data Fabric; Function 4
Product ID: 166e
NUMA node: 0
IOMMU group: 7
Driver:
Device 15:
Address: 0000:00:18.5
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Cezanne Data Fabric; Function 5
Product ID: 166f
NUMA node: 0
IOMMU group: 7
Driver:
Device 16:
Address: 0000:00:18.6
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Cezanne Data Fabric; Function 6
Product ID: 1670
NUMA node: 0
IOMMU group: 7
Driver:
Device 17:
Address: 0000:00:18.7
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Cezanne Data Fabric; Function 7
Product ID: 1671
NUMA node: 0
IOMMU group: 7
Driver:
Device 18:
Address: 0000:01:00.0
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product:
Product ID: 43ec
NUMA node: 0
IOMMU group: 8
Driver: xhci_hcd
Device 19:
Address: 0000:01:00.1
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: 500 Series Chipset SATA Controller
Product ID: 43eb
NUMA node: 0
IOMMU group: 8
Driver: ahci
Device 20:
Address: 0000:01:00.2
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: 500 Series Chipset Switch Upstream Port
Product ID: 43e9
NUMA node: 0
IOMMU group: 8
Driver: pcieport
Device 21:
Address: 0000:02:03.0
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product:
Product ID: 43ea
NUMA node: 0
IOMMU group: 8
Driver: pcieport
Device 22:
Address: 0000:03:00.0
Vendor: Realtek Semiconductor Co., Ltd.
Vendor ID: 10ec
Product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
Product ID: 8168
NUMA node: 0
IOMMU group: 8
Driver: r8169
Device 23:
Address: 0000:04:00.0
Vendor: Sandisk Corp
Vendor ID: 15b7
Product: WD Blue SN570 NVMe SSD 1TB
Product ID: 501a
NUMA node: 0
IOMMU group: 9
Driver: nvme
Device 24:
Address: 0000:05:00.0
Vendor: Advanced Micro Devices, Inc. [AMD/ATI]
Vendor ID: 1002
Product: Cezanne [Radeon Vega Series / Radeon Vega Mobile Series]
Product ID: 1638
NUMA node: 0
IOMMU group: 10
Driver: amdgpu
Device 25:
Address: 0000:05:00.1
Vendor: Advanced Micro Devices, Inc. [AMD/ATI]
Vendor ID: 1002
Product: Renoir Radeon High Definition Audio Controller
Product ID: 1637
NUMA node: 0
IOMMU group: 11
Driver: snd_hda_intel
Device 26:
Address: 0000:05:00.2
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Family 17h (Models 10h-1fh) Platform Security Processor
Product ID: 15df
NUMA node: 0
IOMMU group: 12
Driver: ccp
Device 27:
Address: 0000:05:00.3
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir/Cezanne USB 3.1
Product ID: 1639
NUMA node: 0
IOMMU group: 13
Driver: xhci_hcd
Device 28:
Address: 0000:05:00.4
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Renoir/Cezanne USB 3.1
Product ID: 1639
NUMA node: 0
IOMMU group: 14
Driver: xhci_hcd
Device 29:
Address: 0000:05:00.6
Vendor: Advanced Micro Devices, Inc. [AMD]
Vendor ID: 1022
Product: Family 17h/19h HD Audio Controller
Product ID: 15e3
NUMA node: 0
IOMMU group: 15
Driver: snd_hda_intel
There is a bunch of stuff in IOMMU group 8. Basically devices 18 through 22. Is it either all or nothing?