HowTo Configure RoCE Congestion Control for Windows 2012

Version 17

    This post shows the list of RoCE Congestion Control adapter parameters and configuration commands for a Windows 2012 server using Mellanox ConnectX-3 Pro adapter.

     

     

     

     

    References

     

     

    List of Parameters

     

    Registry Parameter

    Default

    Range

    Description

    EcnEnable

    0

    0/1

    Enables RoCE congestion control

    EcnCnpDscp

    46

    0 – 63

    DSCP value in the IP header of CNP

    EcnCnp802pPrio

    7

    0 - 7

    PCP value in the VLAN header of CNP

    EcnRpgTimeReset

    2000

    0 – u32 [uSec]

    Time counter for rate increase event

    EcnRpgByteReset

    500000

    0 – u32 [Byte]

    Byte counter for rate increase event

    EcnRpgThreshold

    5

    1 – u16

    Number of rate increase events for switching between Fast Recovery, Active Increase, Hyper Active Increase modes.

    EcnRpgAiRate

    5

    1 – line rate [MBit/sec]

    rate increase in AI mode

    EcnRpgHaiRate

    50

    1 – line rate [MBit/sec]

    rate increase in HAI mode

    EcnAlphaToRateShift

    1

    0 - 20

    upon receiving CNP reduce rate
    by (alpha/2^ EcnAlphaToRateShift)

    EcnRpgMinDecFac

    2

    2 - 100

    maximal factor by which the rate can be reduced (2 means that the new rate can be divided by 2 at maximum)

    EcnRpgMinRate

    1

    1 – line rate [bits/sec]

    Minimal rate that can be set

    EcnRateToSetOnFirstCnp

    20000

    1 – line rate [MBit/sec]

    Rate to set to a flow when receives first CNP

    EcnDceTcpG

    16

    0 – 1023 [uSec]

    weight of the new sampling in moving average calculation of alpha

    EcnDceTcpRtt

    40

    0 – u32 [uSec]

    window for sampling of moving average calculation of alpha

    EcnInitialAlphaValue

    512

    0 – 2^16 (fixed point fraction)

    Initial alpha for a flow

    EcnCnpTimer

    40

    0 – u32 [uSec]

    Minimal time between sending consecutive CNPs to the same QP in NP

    EcnCoalesceCnpInRp

    0

    0/1

    Coalesce CNPs in RP, and handle a CNP per flow only once every dceTcpRtt.(contrary to original algorithm)

    EcnClampTgtRate

    0

    0/1

    If set, when receiving a CNP, the target rate is updated to be the current rate. (contrary to original algorithm)

    EcnClampTgtRateAfterTimeInc

    1

    0/1

    If set, when receiving a CNP, the target rate is updated to be the current rate also if the last rate increase event was due to the timer, and not only due to the byte counter (contrary to original algorithm)

    EcnTrapDisablePeriod

    15

    0-1000

    Minimal Time (in microseconds) to disable trapping of ECN marked packets, after trapping an ECN-marked packet.

    Value 0x0 disables the feature

    EcnTrapDisablePendingCnpThreshold

    8

    0-128

    Number of CNPs pending for transmission, above which the trapping of ECN marked packets is disabled.

    Value 0x0 disables the feature

     

     

     

     

    ECN Powershell commands

    1. Get-MlnxNetAdapterEcnSetting

     

    PS C:\Users\Administrator> Get-MlnxNetAdapterEcnSetting -Name "Ethernet 7"

     

    Caption                : MLNX_NetAdapterEcnSettingData 'Mellanox ConnectX-3 Pro Ethernet Adapter'

    Description            : Mellanox ConnectX-3 Pro Ethernet Adapter

    ElementName            : Mellanox ConnectX-3 Pro Ethernet Adapter

    InstanceID             : {47B218B8-B98E-43F5-AD02-7F1210DB5804}

    InterfaceDescription   : Mellanox ConnectX-3 Pro Ethernet Adapter

    Name                   : Ethernet 7

    Source                 : 2

    SystemName             : MTI-MAR-S7

    EcnAlphaToRateShift    :

    EcnCnp802pPrio         :

    EcnCnpDscp             :

    EcnCnpTimer            :

    EcnDceTcpG             :

    EcnDceTcpRtt           :

    EcnEnable              : False

    EcnInitialAlphaValue   :

    EcnRateToSetOnFirstCnp :

    EcnRpgAiRate           :

    EcnRpgByteReset        :

    EcnRpgHaiRate          :

    EcnRpgMinDecFac        :

    EcnRpgMinRate          :

    EcnRpgThreshold        :

    EcnRpgTimeReset        :

    PSComputerName         :

     

     

    2. Enable-MlnxNetAdapterEcnSetting

     

    PS C:\Users\Administrator> Enable-MlnxNetAdapterEcnSetting

     

    Confirm

    Are you sure you want to perform this action?

    Performing the operation "Enable" on target "MLNX_NetAdapterEcnSettingData: MLNX_NetAdapterEcnSettingData 'Mellanox

    ConnectX-3 Pro Ethernet Adapter' (InstanceID = "{47B218B8-B98E-43F5-AD02-7F1210DB5804}")".

    [Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): Y

     

    Confirm

    Are you sure you want to perform this action?

    Performing the operation "Enable" on target "MLNX_NetAdapterEcnSettingData: MLNX_NetAdapterEcnSettingData 'Mellanox

    ConnectX-3 Pro Ethernet Adapter #2' (InstanceID = "{BE5396E6-F654-4A71-8F5B-49F5C687F44A}")".

    [Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): Y

     

     

    PS C:\Users\Administrator> Get-MlnxNetAdapterEcnSetting -Name "Ethernet 7"

     

    Caption                : MLNX_NetAdapterEcnSettingData 'Mellanox ConnectX-3 Pro Ethernet Adapter'

    Description            : Mellanox ConnectX-3 Pro Ethernet Adapter

    ElementName            : Mellanox ConnectX-3 Pro Ethernet Adapter

    InstanceID             : {47B218B8-B98E-43F5-AD02-7F1210DB5804}

    InterfaceDescription   : Mellanox ConnectX-3 Pro Ethernet Adapter

    Name                   : Ethernet 7

    Source                 : 2

    SystemName             : MTI-MAR-S7

    EcnAlphaToRateShift    : 1

    EcnCnp802pPrio         : 0

    EcnCnpDscp             : 0

    EcnCnpTimer            : 20

    EcnDceTcpG             : 64

    EcnDceTcpRtt           : 25

    EcnEnable              : True

    EcnInitialAlphaValue   : 512

    EcnRateToSetOnFirstCnp : 40000

    EcnRpgAiRate           : 10

    EcnRpgByteReset        : 1500000000

    EcnRpgHaiRate          : 50

    EcnRpgMinDecFac        : 2

    EcnRpgMinRate          : 10

    EcnRpgThreshold        : 5

    EcnRpgTimeReset        : 3000

    PSComputerName         :

     

     

    3. Disable-MlnxNetAdapterEcnSetting

    PS C:\Users\Administrator> Disable-MlnxNetAdapterEcnSetting

     

    Confirm

    Are you sure you want to perform this action?

    Performing the operation "Disable" on target "MLNX_NetAdapterEcnSettingData: MLNX_NetAdapterEcnSettingData 'Mellanox

    ConnectX-3 Pro Ethernet Adapter' (InstanceID = "{47B218B8-B98E-43F5-AD02-7F1210DB5804}")".

    [Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y

     

    Confirm

    Are you sure you want to perform this action?

    Performing the operation "Disable" on target "MLNX_NetAdapterEcnSettingData: MLNX_NetAdapterEcnSettingData 'Mellanox

    ConnectX-3 Pro Ethernet Adapter #2' (InstanceID = "{BE5396E6-F654-4A71-8F5B-49F5C687F44A}")".

    [Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): Y

     

     

    4. Invoke-CimMethod

     

    To modify ECN parameters, you should use:

    1. $x=Get-MlnxNetAdapterEcnSetting -Name <name>

    2. Invoke-CimMethod -InputObject $x -MethodName SetValue -Arguments @{<arg>=<value>}

     

    See example below.

     

    ECN Configuration Example

     

    Follow this procedure to to configure ECN:

    1. Enable ECN

    Note: it could be done also per interface using -Name argument)

     

    PS C:\Users\Administrator> Enable-MlnxNetAdapterEcnSetting

     

     

    2. Run the following command:

     

    PS C:\Users\Administrator> $x=Get-MlnxNetAdapterEcnSetting -Name "Ethernet 7"

    PS C:\Users\Administrator> $x

     

    Caption                : MLNX_NetAdapterEcnSettingData 'Mellanox ConnectX-3 Pro Ethernet Adapter'

    Description            : Mellanox ConnectX-3 Pro Ethernet Adapter

    ElementName            : Mellanox ConnectX-3 Pro Ethernet Adapter

    InstanceID             : {47B218B8-B98E-43F5-AD02-7F1210DB5804}

    InterfaceDescription   : Mellanox ConnectX-3 Pro Ethernet Adapter

    Name                   : Ethernet 7

    Source                 : 2

    SystemName             : MTI-MAR-S7

    EcnAlphaToRateShift    : 1

    EcnCnp802pPrio         : 0

    EcnCnpDscp             : 0

    EcnCnpTimer            : 20

    EcnDceTcpG             : 64

    EcnDceTcpRtt           : 25

    EcnEnable              : True

    EcnInitialAlphaValue   : 512

    EcnRateToSetOnFirstCnp : 40000

    EcnRpgAiRate           : 10

    EcnRpgByteReset        : 1500000000

    EcnRpgHaiRate          : 50

    EcnRpgMinDecFac        : 2

    EcnRpgMinRate          : 10

    EcnRpgThreshold        : 5

    EcnRpgTimeReset        : 3000

    PSComputerName         :

     

    3. To change a specific parameter run:

     

    In this case, we are changing EcnCnp802pPrio parameter to 3.

    PS C:\Users\Administrator> Invoke-CimMethod -InputObject $x -MethodName SetValue -Arguments @{EcnCnp802pPrio=3}
    PCComputerName
    --------------

     

    Another example is to change ECNCnpDscp to 34.

    PS C:\Users\Administrator> Invoke-CimMethod -InputObject $x -MethodName SetValue -Arguments @{EcnCnpDscp=34}
    PCComputerName
    --------------

     

    4. Verify that the change took place, run:

    PS C:\Users\Administrator> Get-MlnxNetAdapterEcnSetting -Name "Ethernet 7"

     

    Caption                : MLNX_NetAdapterEcnSettingData 'Mellanox ConnectX-3 Pro Ethernet Adapter'

    Description            : Mellanox ConnectX-3 Pro Ethernet Adapter

    ElementName            : Mellanox ConnectX-3 Pro Ethernet Adapter

    InstanceID             : {47B218B8-B98E-43F5-AD02-7F1210DB5804}

    InterfaceDescription   : Mellanox ConnectX-3 Pro Ethernet Adapter

    Name                   : Ethernet 7

    Source                 : 2

    SystemName             : MTI-MAR-S7

    EcnAlphaToRateShift    : 1

    EcnCnp802pPrio         : 3

    EcnCnpDscp             : 34

    EcnCnpTimer            : 20

    EcnDceTcpG             : 64

    EcnDceTcpRtt           : 25

    EcnEnable              : True

    EcnInitialAlphaValue   : 512

    EcnRateToSetOnFirstCnp : 40000

    EcnRpgAiRate           : 10

    EcnRpgByteReset        : 1500000000

    EcnRpgHaiRate          : 50

    EcnRpgMinDecFac        : 2

    EcnRpgMinRate          : 10

    EcnRpgThreshold        : 5

    EcnRpgTimeReset        : 3000

    PSComputerName         :

     

    Monitoring Counters

    Monitoring counters can be found using perfmon tool within the Mellanox Adapter QoS Counters

     

    7.PNG