Resilient RoCE Configuration for WinOF2 driver in DSCP-based QoS mode

Version 2

    The following post shows a configuration example of RoCE  for Mellanox Adapters over MS-Windows based host for DSCP-based resilient mode.

    For other RoCE configuration examples, see Getting Started with RoCE Configuration .

     

     

     

    Overview

    Assumptions and requirements:

     

    Configuration

    1. Install DCB module, enable DCB in willing mode and remove all previous QoS configuration

    Install-WindowsFeature “data-center-bridging”

    Remove-NetQosTrafficClass

    Remove-NetQosPolicy -Confirm:$False

    Set-NetAdapterQos -Enabled 1 *

    Set-NetQosDcbxSetting -Willing 0 -Confirm:$false

     

    2. Set the QoS rules, mapping SMB to priority 3 (RoCE)

    New-NetQosPolicy “SMB"  -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3  -PolicyStore activestore -Confirm:$false

    New-NetQosPolicy “SMB"  -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3  -Confirm:$false

    New-NetQosPolicy “DEFAULT"  -PriorityValue8021Action 3 -PolicyStore  activestore -Confirm:$false -Default

    New-NetQosPolicy “DEFAULT" -PriorityValue8021Action 3 -Confirm:$false -Default

    New-NetQosPolicy “TCP"  -IPProtocolMatchCondition TCP -PriorityValue8021Action 0 -DSCPAction 0 -PolicyStore activestore -Confirm:$false

    New-NetQosPolicy “TCP"  -IPProtocolMatchCondition TCP -PriorityValue8021Action 0 -DSCPAction 0 -Confirm:$false

    New-NetQosPolicy “UDP"  -IPProtocolMatchCondition UDP -PriorityValue8021Action 0 -PolicyStore activestore -Confirm:$false

    New-NetQosPolicy “UDP"  -IPProtocolMatchCondition UDP -PriorityValue8021Action 0 -Confirm:$false

     

    3. Activate flow control on priority 3.

    Disable-NetQosFlowControl 0,1,2,3,4,5,6,7

    Note: This step can be waived if switch configure the PFC via LLDPs

     

    4. Enable DC-QCN on priority 3 and set DSCP value for CNPs to 48.

    Mlx5Cmd.exe -Qosconfig -Dcqcn -Name "Ethernet" -Set –DcqcnEnablePrio3 1 -DcqcnCnpDscp 48 –DcqcnCnpPrioMode 0

     

    5. Locate the registry key for the Mellanox adapter, see HowTo Locate the Windows Registry key for Mellanox Adapters .

     

    6. Use trust layer 3 (DSCP) mode by setting of RxTrustedState to '2'. (For trust layer 2 (PCP), value of '1' is used )

    PS $  new-itemProperty -Path HKLM:\SYSTEM\CurrentControlSet\Control\Class\"{4d36e972-e325-11ce-bfc1-08002be10318}"\0002\ -Name "RxTrustedState" -PropertyType "String" -Value "2" -Force

     

    7. Set DSCP-based PFC (value of '1'). (The default is PCP-based PFC (value '0') .

    PS $  new-itemProperty -Path HKLM:\SYSTEM\CurrentControlSet\Control\Class\"{4d36e972-e325-11ce-bfc1-08002be10318}"\0002\ -Name "TxUntagPriorityTag" -PropertyType "String" -Value "1" Force

     

    8. Set DSCP value 26 for priority 3 (RoCE traffic)

    PS $  new-itemProperty -Path HKLM:\SYSTEM\CurrentControlSet\Control\Class\"{4d36e972-e325-11ce-bfc1-08002be10318}"\0003\ -Name "PriorityToDscpMappingTable_3" -PropertyType "String" -Value "26" -Force