1 Reply Latest reply on Mar 20, 2018 5:41 AM by samerka

    Unable to configure SR-IOV on Connect-IB



      I have followed all the instruction in this article.HowTo Configure SR-IOV for Connect-IB/ConnectX-4 with KVM (InfiniBand)   but I am getting following error while Setting the desired number of VFs using "echo 6 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs"

      -bash: echo: write error: Invalid argument

      Any help would be greatly appreciated.


      intel_iommu=on and iommu=pt are added to kernel bootime parameter


      # cat /proc/cmdline

      BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.el7.x86_64 root=UUID=51777676-1b13-40a7-aed8-12e9609e4b31 ro intel_pstate=disable console=tty0 console=ttyS0,115200n8 net.ifnames=0 crashkernel=auto rhgb quiet intel_iommu=on iommu=pt




      mlxconfig -d /dev/mst/mt4113_pciconf0 q


      Device #1:



      Device type:    ConnectIB      

      PCI device:     /dev/mst/mt4113_pciconf0


      Configurations:                              Next Boot

               ROCE_NEXT_PROTOCOL                  254            

               NUM_OF_VFS                          6              

               SRIOV_EN                            True(1)   





        • Re: Unable to configure SR-IOV on Connect-IB



          Kindly note that SR-IOV protocol need to be enabled in 4 different places:

          1. Firmware level - according to mlxconfig output it is enabled.

          2. System BIOS - need to verify the "Virtualization Technology" option is enabled.

          3. Operation System at grub.conf - we can see that "intel_iommu=on" exists

          In Connect-IB you must also set FPP_EN=1

          4. Driver - Set the desired number of VFs by invoking:

          echo 4 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs

          cat /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs


          5. The command to enable SRIOV support for ConnectX4 , Connect-IB and ConnectX-5 on an MLNX-OS based subnet manage is:

          switch(config)# ib sm virt enable


          In addition, if after applying the above settings you still encounter the same issue

          I suggest reviewing the release notes of the latest OFED 4.3 : http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_Release_Notes_4_3-1_0_1_0.pdf

          And check if there are known issues with SRIOV with older OFED versions , if yes please try and upgrade the OFED version accordingly and check if the issue resolved .


          6. Try using the latest MFT 4.9.0-38 and try configuring FPP_EN.