HowTo Configure SR-IOV for ConnectX-3 with Hyper-V (Ethernet)

Version 10

    This post shows how to configure SR-IOV over Hyper-V with ConnectX-3/Pro in Windows 2012 R2 environment for Ethernet networks.

     

    References

     

    Note: In case SR-IOV is enabled on the adapter, RDMA is disabled.

     

    Configuration

     

    1. Make sure that Virtualization and SR-IOV Global Enable are enabled in the BIOS. Refer to the BIOS manual of the server manufacturer.

        In the following example, we will set 8 VMs on one port and 8 VMs on the second adapter port. Make sure to enable the SR-IOV on the Mellanox adapter (per port) in the BIOS.

       Here is an example of Dell R730 BIOS configuration HowTo Set Dell PowerEdge R730 BIOS parameters to support SR-IOV.

     

    2. Install Windows Server 2012 R2 and drivers.

    • Install the latest WinOF driver.
    • Make sure that the Firmware for the card is aligned to the WinOF driver. Refer to the WinOF driver Release notes for supported Fimware.

     

    3. Install Hyper-V Role on the server. Click on Server Manager -> Manage- Add Roles and Features and set the following:

    • Installation Type -> Role-based or Feature-based Installation
    • Server Selection -> Select a server from the server pool
    • Server Roles -> Hyper-V

    hyper-v-1.png

     

    4. Install Hyper-V Administration Tools.

     

    Features - > Remote Server Administration Tools -> Role Administration Tools -> Hyper-V Administration Tool

    hyper-v-2.png

     

    5. Do not select the Mellanox Adapter for the virtual switch configuration. It will be done later.

    Mellanox card.JPG

     

    6. Confirm the installation and reboot the system.

     

    7. Verify that SR-IOV is supported

    PS C:\Users\Administrator> (Get-VmHost).IovSupport

    True

    PS C:\Users\Administrator> (Get-VmHost).IovSupportReasons

    OK

    PS C:\Users\Administrator> Get-MlnxPCIDeviceSriovSetting

     

    Note: At this point SR-IOV is supported, but not enabled on the driver.

    PS C:\Users\Administrator> Get-MlnxPCIDeviceSriovSetting

     

    Caption          : MLNX_PCIDeviceSriovSettingData 'Mellanox ConnectX-3 PRO VPI (MT04103) Network Adapter'

    Description      : Mellanox ConnectX-3 PRO VPI (MT04103) Network Adapter

    ElementName      : HCA 0

    InstanceID       : PCI\VEN_15B3&DEV_1007&SUBSYS_008015B3&REV_00\E41D2D03001FEDC000

    Name             : HCA 0

    Source           : 3

    SystemName       : WIN-VAV4CVO1FP6

    SriovEnable      : False

    SriovPort1NumVFs : 16

    SriovPort2NumVFs : 0

    SriovPortMode    : 0

    PSComputerName   :

     

    8. Check that the Hyper-V Server Manager was added to the dashboard.

    Hyper-V Server Manager.JPG

     

    9. Set the ExecutionPolicy to all signed. This step is not relevant specifically to SR-IOV, but to all scripts. The OS will not allow any scripts, unless setting the ExecutionPolicy to AllSigned.

    PS C:\Users\Administrator> Set-ExecutionPolicy AllSigned

     

    Execution Policy Change

    The execution policy helps protect you from scripts that you do not trust. Changing the execution policy might expose

    you to the security risks described in the about_Execution_Policies help topic at

    http://go.microsoft.com/fwlink/?LinkID=135170. Do you want to change the execution policy?

    [Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"): Y

     

    10. Enable SR-IOV on the driver.

          In this example, we enable SR-IOV and set 8 VMs on the first port and 8 VMs on the second port.

    PS C:\Users\Administrator> Set-MlnxPCIDeviceSriovSetting -Name "HCA 0" -SriovEnable $true -SriovPortMode 2 -SriovPort1Nu

    mVFs 8 -SriovPort2NumVFs 8

     

    Confirm

    Are you sure you want to perform this action?

    Performing the operation "SetValue" on target "MLNX_PCIDeviceSriovSettingData: MLNX_PCIDeviceSriovSettingData 'Mellanox

    ConnectX-3 PRO VPI (MT04103) Network Adapter' (InstanceID = "PCI\VEN_15B3&DEV_1007&SUBSYS_008015B3&R...)".

    [Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): Y

     

    PS C:\Users\Administrator> Get-MlnxPCIDeviceSriovSetting

     

    Caption          : MLNX_PCIDeviceSriovSettingData 'Mellanox ConnectX-3 PRO VPI (MT04103) Network Adapter'

    Description      : Mellanox ConnectX-3 PRO VPI (MT04103) Network Adapter

    ElementName      : HCA 0

    InstanceID       : PCI\VEN_15B3&DEV_1007&SUBSYS_008015B3&REV_00\E41D2D03001FEDC000

    Name             : HCA 0

    Source           : 3

    SystemName       : WIN-VAV4CVO1FP6

    SriovEnable      : True

    SriovPort1NumVFs : 8

    SriovPort2NumVFs : 8

    SriovPortMode    : 2

    PSComputerName   :

     

    Note: Make sure that that number shown here is aligned with the number configured in the BIOS.  In case the BIOS is configured with 8 VMs, while the adapter is enabling 16 VMs, you will get only 8 VMs in total, aligned with the BIOS.

     

    11. Create a virtual machine:

          Go to Server Manager->Tools-> Hyper V Manager and click on New Virtual Machine.

    vm-1.PNG

     

    12. Make sure to set 4096MB or more for the Memory.

     

    memory.PNG

     

    Note:  The OS can be installed on the Virtual Machine after creating the Virtual Machine.

     

    13 . Create a virtual switch:

           Go to: Hyper-V Manager -> Action->Virtual Switch Manager -> External

     

    14. Rename the switch name (In this case, Mellanox Virtual Switch), select the relevant Adapter port and enable SR-IOV.

    Virtual Switch.JPG

     

    15. Add VMNIC to the VM.

          Under Actions, go to Settings -> Add New Hardware -> Network Adapter -> OK.

          In "Virtual switch" dropdown box, Choose Mellanox Virtual Switch.

    VMNIC to VSwitch.JPG

     

     

    16. Enable the SR-IOV for Mellanox VMNIC:

    Hardware Acc - SRIOV.JPG

    17. Start and Connect to the Virtual machine. Install the OS which can be Windows Server 2012 and above.

     

    18. Install the WinOF driver on the VM.

     

          Note: In order to ping between the Hypervisor and the Virtual Machine you need to disable the Firewall.

     

    19. Reboot the Virtual Machine once the OS is installed on the Virtual Machine.

     

    20. Verify that Mellanox Virtual Function appears in the Device Manager.

     

    Device Manager in VM.JPG