HowTo Configure CoS and DSCP for SMB in Windows 2012

Version 12

    This post discusses various considerations regarding QoS configuration for ConnectX-3 / ConnectX-3 Pro adapters in a Windows 2012 environment.

    This post is for advanced users and meant for IT architects and Windows experts.

     

    References

     

    Setup

    • Mellanox ConnectX-3 / ConnectX-3 Pro adapters
    • WinOF 4.95 Drivers
    • 40GbE switch

     

    Overview

    DSCP (0...63) and CoS (0...7) are related and have default mapping in case of SMB Direct (RDMA) traffic type.

     

    The default mapping is 1-to-1. For example CoS 0 is mapped to DSCP 0, CoS 1 is mapped to DSCP 1, ..., and CoS 7 is mapped to DSCP 7.

    In case you wish to have a different DSCP mapping for RDMA traffic, follow the examples below.

     

     

    Example 1: How to mark CoS 4 and DSCP 34 for SMB traffic over RoCEv2

     

    1. Make sure SMB can run over RoCEv2 between two servers. Follow HowTo Configure SMB Direct over IP networks (RoCEv2) on Windows 2012 Server to verify this.

     

    2. Create a QoS policy to map SMB traffic with CoS 4

     

    PS C:\Users\Administrator> New-NetQosPolicy “Policy All CoS 4” -PolicyStore ActiveStore –NetDirectPort 445 -PriorityValue8021Action 4

    Name           : policy all cos 4

    Owner          : PowerShell / WMI

    NetworkProfile : All

    Precedence     : 127

    NetDirectPort  : 445

    PriorityValue  : 4

    3. Open regedit and reach the driver key of the specific port. Follow HowTo Locate the Windows Registry key for Mellanox Adapters for instructions.

     

    4. Right click to add a new String Value parameter (if it does not exist already) called PriorityToDscpMappingTable_4 with value 34.

    reg3.png

     

     

    reg4.PNG

     

    Note: This is relevant only for RDMA traffic

     

    Another way to set the parameter is via the following powershell command:

    PS C:\Users\Administrator> new-itemProperty -Path HKLM:\SYSTEM\CurrentControlSet\Control\Class\"{4d36e972-e325-11ce-bfc1-08002be10318}"\0010\ -Name "PriorityToDscpMappingTable_4" -PropertyType "String" -Value "32" -Force

     

     

    Note that in this case the key \"{4d36e972-e325-11ce-bfc1-08002be10318}"\ is fixed for all Mellanox adapter cards.

     

    It is possible to get the driver key + if index via this script  (change the "Etherent" port name to match your setup):

    PS C:\Users\Administrator> $b = (Get-NetAdapter -Name "Ethernet").InterfaceDescription; $a = ((Get-Device | Where-Object {($_.Name -eq $b)}).AvailableProperties); $a.DEVPKEY_Device_Driver

    {4d36e972-e325-11ce-bfc1-08002be10318}\0010

    PS C:\Users\Administrator>

     

    Here is a full script:

    PS C:\Users\Administrator>  $b = (Get-NetAdapter -Name "Ethernet").InterfaceDescription; $a = ((Get-Device | Where-Object {($_.Name -eq $b)}).AvailableProperties); $c = “HKLM:\SYSTEM\CurrentControlSet\Control\Class\” + $a.DEVPKEY_Device_Driver; new-itemProperty -Path $c -Name "PriorityToDscpMappingTable_4" -PropertyType "String" -Value "32" -Force

     

    PriorityToDscpMappingTable_4 : 32

    PSPath                       : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\

                                   Class\{4d36e972-e325-11ce-bfc1-08002be10318}\0010

    PSParentPath                 : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\

                                   Class\{4d36e972-e325-11ce-bfc1-08002be10318}

    PSChildName                  : 0010

    PSDrive                      : HKLM

    PSProvider                   : Microsoft.PowerShell.Core\Registry

     

    5. Reset the adapter port.

    You can do that by disabling and enabling from the Driver tab of the port's Adapter Properties dialog.

     

     

    disable.PNG

     

    6. Run some SMB Direct over RoCeV2 traffic and capture the frames on the switch.

     

    You should expect to receive UDP frames (RoCEv2 is running over UDP frames) with CoS 4 (PRI=4) and DSCP 34 (0x22h).

    Here is a wireshark snapshot.

    ws_1.PNG

     

    Example 2: How to mark two types of traffic with different CoS values

     

    • SMB traffic over RoCEv2: CoS 4 and DSCP 34
    • TCP traffic: CoS 3 and DSCP 26

     

    1. Follow the procedure in Example 1 above.

     

    2. Create another QoS policy to map TCP traffic with CoS 3 and DSCP 26.

    PS C:\Users\Administrator> New-NetQosPolicy "Policy TCP CoS 4 DSCP 26" -IPProtocolMatchCondition TCP -PriorityValue8021Action 3 -DSCPAction 26 -PolicyStore ActiveStore

    Name           : policy tcp cos 4 dscp 26

    Owner          : PowerShell / WMI

    NetworkProfile : All

    Precedence     : 127

    IPProtocol     : TCP

    PriorityValue  : 3

    DSCPValue      : 26

     

    3. Run some SMB Direct over RoCeV2 traffic and TCP traffic and capture the frames on the switch.

     

    You should expect to receive:

    • UDP frames (RoCEv2 is running over UDP frames) with CoS 4 (PRI=4) and DSCP 34 (0x22h)
    • TCP frames with CoS 3 (PRI=3) and DSCP 26 (0x1Ah)

     

    Here is a wireshark snapshot example for the TCP traffic.

    tcp traffic cos 3_marked.PNG