Trying to use SR-IOV with Mellanox Infiniband: All virtual functions on device "sriov" are already in use

havan · July 28, 2018, 2:50am

I was trying to use SR-IOV device with infiniband device type. I get the error below and couldn’t get any more info:

 [root@lx16 ~]# lxc start computenode                                                                                                  
Error: All virtual functions on device "sriov" are already in use
Try `lxc info --show-log computenode` for more info
 [root@lx16 ~]# lxc info --show-log computenode
Name: computenode
Location: lx16
Remote: unix://
Architecture: x86_64
Created: 2018/07/27 23:40 UTC
Status: Stopped
Type: persistent
Profiles: default, compute, sr-iov
Snapshots:

Log:

 [root@lx16 ~]# lxc profile show sr-iov
config: {}
description: ""
devices:
  sriov:
    nictype: sriov
    parent: ib0
    type: infiniband
name: sr-iov
used_by:
- /1.0/containers/computenode

 [root@lx16 ~]# lspci|grep -i mell
b0:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
b0:00.1 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b0:00.2 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

stgraber · July 28, 2018, 3:14am

Can you show:

ls -lh /sys/class/net/ib0/device/
ls -lh /sys/class/net/ib0/device/virtfn*/
ls -lh /sys/class/net/ib0/device/virtfn*/net/

LXD will only consider virtual functions that have network devices listed under net/, otherwise it’s assuming that they’re already in another network namespace and so considered to be in use.

stgraber · July 28, 2018, 3:15am

Ah and the values for:

cat /sys/class/net/ib0/device/sriov_numvfs
cat /sys/class/net/ib0/device/sriov_totalvfs

Would be good to check too, in your case I’d expect it to be 2 in both cases, but worth making sure.

stgraber · July 28, 2018, 3:17am

It could also be that you have VFs enabled but don’t have them bound to the IB driver, you can find more details on how I got things going here:

Though that was with ConnectX-4 so the module names may differ in your case.

havan · July 28, 2018, 1:07pm

/sys/class/net/ib0/device/:

# ls -lh /sys/class/net/ib0/device/
total 0
-rw-r--r-- 1 root root 4,0K Tem 28 15:52 broken_parity_status
-r--r--r-- 1 root root 4,0K Tem 28 15:52 class
-rw-r--r-- 1 root root 4,0K Tem 28 15:52 config
-r--r--r-- 1 root root 4,0K Tem 28 15:52 consistent_dma_mask_bits
-rw-r--r-- 1 root root 4,0K Tem 28 15:52 d3cold_allowed
-r--r--r-- 1 root root 4,0K Tem 28 15:52 device
-r--r--r-- 1 root root 4,0K Tem 28 15:52 dma_mask_bits
lrwxrwxrwx 1 root root    0 Tem 28 15:52 driver -> ../../../../bus/pci/drivers/mlx4_core
-rw-r--r-- 1 root root 4,0K Tem 28 15:52 driver_override
-rw-r--r-- 1 root root 4,0K Tem 28 15:52 enable
drwxr-xr-x 3 root root    0 Tem 28 15:52 infiniband
drwxr-xr-x 3 root root    0 Tem 28 15:52 infiniband_cm
drwxr-xr-x 4 root root    0 Tem 28 15:52 infiniband_mad
drwxr-xr-x 3 root root    0 Tem 28 15:52 infiniband_verbs
lrwxrwxrwx 1 root root    0 Tem 28 15:52 iommu -> ../../../virtual/iommu/dmar0
lrwxrwxrwx 1 root root    0 Tem 28 15:52 iommu_group -> ../../../../kernel/iommu_groups/30
-r--r--r-- 1 root root 4,0K Tem 28 15:52 irq
-r--r--r-- 1 root root 4,0K Tem 28 15:52 local_cpulist
-r--r--r-- 1 root root 4,0K Tem 28 15:52 local_cpus
-r--r--r-- 1 root root 4,0K Tem 28 15:52 mlx4_port1
-r--r--r-- 1 root root 4,0K Tem 28 15:52 mlx4_port1_mtu
-r--r--r-- 1 root root 4,0K Tem 28 15:52 modalias
-rw-r--r-- 1 root root 4,0K Tem 28 15:52 msi_bus
drwxr-xr-x 2 root root    0 Tem 28 15:52 msi_irqs
drwxr-xr-x 3 root root    0 Tem 28 15:52 net
-rw-r--r-- 1 root root 4,0K Tem 28 15:52 numa_node
-r--r--r-- 1 root root 4,0K Tem 28 15:52 pools
drwxr-xr-x 2 root root    0 Tem 28 15:52 power
--w--w---- 1 root root 4,0K Tem 28 15:52 remove
--w--w---- 1 root root 4,0K Tem 28 15:52 rescan
--w------- 1 root root 4,0K Tem 28 15:52 reset
-r--r--r-- 1 root root 4,0K Tem 28 15:52 resource
-rw------- 1 root root 1,0M Tem 28 15:52 resource0
-rw------- 1 root root 8,0M Tem 28 15:52 resource2
-rw------- 1 root root 8,0M Tem 28 15:52 resource2_wc
-rw-rw-r-- 1 root root 4,0K Tem 28 15:52 sriov_numvfs
-r--r--r-- 1 root root 4,0K Tem 28 15:52 sriov_totalvfs
lrwxrwxrwx 1 root root    0 Tem 28 15:52 subsystem -> ../../../../bus/pci
-r--r--r-- 1 root root 4,0K Tem 28 15:52 subsystem_device
-r--r--r-- 1 root root 4,0K Tem 28 15:52 subsystem_vendor
-rw-r--r-- 1 root root 4,0K Tem 28 15:52 uevent
-r--r--r-- 1 root root 4,0K Tem 28 15:52 vendor
lrwxrwxrwx 1 root root    0 Tem 28 15:52 virtfn0 -> ../0000:b0:00.1
lrwxrwxrwx 1 root root    0 Tem 28 15:52 virtfn1 -> ../0000:b0:00.2
-rw------- 1 root root  32K Tem 28 15:52 vpd

/sys/class/net/ib0/device/virtfn*/:

# ls -lh /sys/class/net/ib0/device/virtfn*/
/sys/class/net/ib0/device/virtfn0/:
total 0
-rw-r--r-- 1 root root 4,0K Tem 28 15:53 broken_parity_status
-r--r--r-- 1 root root 4,0K Tem 28 15:53 class
-rw-r--r-- 1 root root 4,0K Tem 28 15:53 config
-r--r--r-- 1 root root 4,0K Tem 28 15:53 consistent_dma_mask_bits
-rw-r--r-- 1 root root 4,0K Tem 28 15:53 d3cold_allowed
-r--r--r-- 1 root root 4,0K Tem 28 15:53 device
-r--r--r-- 1 root root 4,0K Tem 28 15:53 dma_mask_bits
-rw-r--r-- 1 root root 4,0K Tem 28 15:53 driver_override
-rw-r--r-- 1 root root 4,0K Tem 28 15:53 enable
lrwxrwxrwx 1 root root    0 Tem 28 15:53 iommu -> ../../../virtual/iommu/dmar0
lrwxrwxrwx 1 root root    0 Tem 28 15:53 iommu_group -> ../../../../kernel/iommu_groups/42
-r--r--r-- 1 root root 4,0K Tem 28 15:53 irq
-r--r--r-- 1 root root 4,0K Tem 28 15:53 local_cpulist
-r--r--r-- 1 root root 4,0K Tem 28 15:53 local_cpus
-r--r--r-- 1 root root 4,0K Tem 28 15:53 modalias
-rw-r--r-- 1 root root 4,0K Tem 28 15:53 msi_bus
-rw-r--r-- 1 root root 4,0K Tem 28 15:53 numa_node
lrwxrwxrwx 1 root root    0 Tem 28 15:53 physfn -> ../0000:b0:00.0
drwxr-xr-x 2 root root    0 Tem 28 15:53 power
--w------- 1 root root 4,0K Tem 28 15:53 reset
-r--r--r-- 1 root root 4,0K Tem 28 15:53 resource
-rw------- 1 root root 8,0M Tem 28 15:53 resource2
-rw------- 1 root root 8,0M Tem 28 15:53 resource2_wc
lrwxrwxrwx 1 root root    0 Tem 28 15:53 subsystem -> ../../../../bus/pci
-r--r--r-- 1 root root 4,0K Tem 28 15:53 subsystem_device
-r--r--r-- 1 root root 4,0K Tem 28 15:53 subsystem_vendor
-rw-r--r-- 1 root root 4,0K Tem 28 15:53 uevent
-r--r--r-- 1 root root 4,0K Tem 28 15:53 vendor

/sys/class/net/ib0/device/virtfn1/:
total 0
-rw-r--r-- 1 root root 4,0K Tem 28 15:54 broken_parity_status
-r--r--r-- 1 root root 4,0K Tem 28 15:54 class
-rw-r--r-- 1 root root 4,0K Tem 28 15:54 config
-r--r--r-- 1 root root 4,0K Tem 28 15:54 consistent_dma_mask_bits
-rw-r--r-- 1 root root 4,0K Tem 28 15:54 d3cold_allowed
-r--r--r-- 1 root root 4,0K Tem 28 15:54 device
-r--r--r-- 1 root root 4,0K Tem 28 15:54 dma_mask_bits
-rw-r--r-- 1 root root 4,0K Tem 28 15:54 driver_override
-rw-r--r-- 1 root root 4,0K Tem 28 15:54 enable
lrwxrwxrwx 1 root root    0 Tem 28 15:54 iommu -> ../../../virtual/iommu/dmar0
lrwxrwxrwx 1 root root    0 Tem 28 15:54 iommu_group -> ../../../../kernel/iommu_groups/43
-r--r--r-- 1 root root 4,0K Tem 28 15:54 irq
-r--r--r-- 1 root root 4,0K Tem 28 15:54 local_cpulist
-r--r--r-- 1 root root 4,0K Tem 28 15:54 local_cpus
-r--r--r-- 1 root root 4,0K Tem 28 15:54 modalias
-rw-r--r-- 1 root root 4,0K Tem 28 15:54 msi_bus
-rw-r--r-- 1 root root 4,0K Tem 28 15:54 numa_node
lrwxrwxrwx 1 root root    0 Tem 28 15:54 physfn -> ../0000:b0:00.0
drwxr-xr-x 2 root root    0 Tem 28 15:54 power
--w------- 1 root root 4,0K Tem 28 15:54 reset
-r--r--r-- 1 root root 4,0K Tem 28 15:54 resource
-rw------- 1 root root 8,0M Tem 28 15:54 resource2
-rw------- 1 root root 8,0M Tem 28 15:54 resource2_wc
lrwxrwxrwx 1 root root    0 Tem 28 15:54 subsystem -> ../../../../bus/pci
-r--r--r-- 1 root root 4,0K Tem 28 15:54 subsystem_device
-r--r--r-- 1 root root 4,0K Tem 28 15:54 subsystem_vendor
-rw-r--r-- 1 root root 4,0K Tem 28 15:54 uevent
-r--r--r-- 1 root root 4,0K Tem 28 15:54 vendor

ls -lh /sys/class/net/ib0/device/virtfn*/net/:

# ls -lh /sys/class/net/ib0/device/virtfn*/net/
ls: cannot access '/sys/class/net/ib0/device/virtfn*/net/': No such file or directory

LXD will only consider virtual functions that have network devices listed under net/, otherwise it’s assuming that they’re already in another network namespace and so considered to be in use.

So it seems that I do not have.

havan · July 28, 2018, 1:10pm

Yes, both of these returns 2.

havan · July 28, 2018, 1:12pm

If they were not bound to the driver, do they show up in lspci output?

# lspci|grep -i mell
b0:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
b0:00.1 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
b0:00.2 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

stgraber · July 28, 2018, 4:18pm

Your output above is consistent with you having VFs enabled but not bound to the driver.
PCI devices will show up regardless of whether a driver is using them or not.

In the post I referred to, you can see me load the driver with both probe_vf and num_vfs set to the number of VFs. The probe_vf property is what instructs the driver to bind to the VFs, only setting num_vfs would get you what you showed above.