1 Reply Latest reply on May 31, 2017 11:46 PM by vitaliyra

    Windows Server 2016 configure RDMA for S2D

    next40

      Hi all, i have configured cluster S2D , two nodes with rdma adapters (HP 546SFP+ ) connected to Switch Dell PowerConnect 8024 with lastest firmware.

      Cluster have two networks : 1 routable for VMs, 2 non routable for Storage RDMA

      I use this PS code for configure RDMA on nodes.....

       

      Install-WindowsFeature Data-Center-Bridging
      Set-NetQosDcbxSetting -Willing $False
      Enable-NetAdapterRdma –Name RDMA-NIC1
      Remove-NetQosTrafficClass -confirm:$False
       Remove-NetQosPolicy -confirm:$False
       New-NetQosPolicy "SMBDIRECT" -netDirectPortMatchCondition 445 -PriorityValue8021Action 4 
      New-NetQosPolicy "DEFAULT" -default  -PriorityValue8021Action 1
      Enable-NetQosFlowControl -Priority 4 
      Disable-NetQosFlowControl 0,1,2,3,5,6,7
       Enable-NetAdapterQos -InterfaceAlias "RDMA-NIC1" 
      New-NetQoSTrafficClass "SMBDirect" -Priority 4 -Bandwidth 90 -Algorithm ETS
      

       

      Flow control disabled on NIC

      GlobalPause disabled on NIC

      VlanID set to 666

       

      Switch config:

      Jumbo Frames enabled

      flowcontrol globally disabled(global pause)

      Cut-through mode is configured as enabled

      Ports config:

      interface Te1/0/20
      datacenter-bridging
      priority-flow-control mode on
      priority-flow-control priority 4 no-drop
      exit
      spanning-tree portfast
      mtu 9216
      switchport mode trunk
      exit
      
      interface Te1/0/10
      datacenter-bridging
      priority-flow-control mode on
      priority-flow-control priority 4 no-drop
      exit
      spanning-tree portfast
      mtu 9216
      switchport mode trunk
      exit
      

      After configuration i see RDMA activity counters greater than zero.... RDMA works, but  i see Mellanox Diagnostics counters  have many errors..... screen shot attached

      nd_send_bw test results:

       

      C:\Program Files\Mellanox\MLNX_VPI\IB\Tools>nd_send_bw -C 10.100.100.2

      #qp #bytes #iterations    MR [Mmps]     Gb/s     CPU Util.

      0   65536     100000       0.017        9.13     100.00

      Test finished. Releasing resources...

       

      Can i fix errors or this is correct RDMA work? Can verify RDMA works correctly?

      thx for help......