HowTo Tune Receive Buffers on Mellanox Adapter Cards

Version 16

    This post discusses the parameters required to tune the Receive Buffer configuration on Mellanox adapter in Ethernet mode.

    This post is meant for advanced technical network engineers, and can be applied on MLNX_OFED v4.2 and above.

     

    References

     

    Overview

    This post explains how to use Linux driver to set specific Receive Buffer sizes and map priorities to Receive Buffer.

    Note: Changes should be performed carefully as they will affect traffic.

     

    Parameters

    buffer_size

    This parameter is used to set the buffer size

    1. To query the Receive Buffer size, run the following command:

    # cat /sys/class/net/eth1/qos/buffer_size

    Port buffer size = 262016

    Spare buffer size = 128

    Buffer Size

    0 130944

    1 130944

    2 0

    3 0

    4 0

    5 0

    6 0

    7 0

    Note: To understand which priority belongs to which buffer, see prio2buffer parameter below.

     

    2. To change the Receive Buffer size, echo an array of sizes to the buffer_size parameter. For example:

    # echo 30000,20000,40000,30000,30000,40000,20000,30000 > /sys/class/net/eth1/qos/buffer_size

    # cat /sys/class/net/eth1/qos/buffer_size

    Port buffer size = 262016

    Spare buffer size = 22400

    Buffer Size

    0 29952

    1 19968

    2 39936

    3 29952

    4 29952

    5 39936

    6 19968

    7 29952

     

    prio2buffer

    This parameter shows the mapping between priority to buffer.

    In the example below, all priorities except for priority 3 go to buffer 0:

    # cat /sys/class/net/eth1/qos/prio2buffer

    Priority Buffer

    0 0

    1 0

    2 0

    3 1

    4 0

    5 0

    6 0

    7 0

    You can manually change this mapping:

    # echo 0,1,2,3,4,1,6,5 > /sys/class/net/eth1/qos/prio2buffer

    # cat /sys/class/net/eth1/qos/prio2buffer

    Priority Buffer

    0 0

    1 1

    2 2

    3 3

    4 4

    5 1

    6 6

    7 5

     

    xon/xoff thresholds (via cable length)

    The driver does not expose the control interface for xon/xoff thresholds, but automatically sets the threshold values via the cable length, port speed and MTU. The default cable length is 7 meters.

    Formula:

    xoff_threshold = buffer size - ((301+2.16 * len [m]) * max(speed [Gbps], 40Gbps) + 2.72 MTU [B])

    xon_threshold = xoff_threshold - 9216

     

    Note: For lossless buffer, make sure to allocate an adequate buffer size (per speed, MTU and cable length).

    The cable length can be extracted by using mlxcables tool. For further information, see HowTo Find Cable Info on Mellanox Adapters and Switches.

     

    To set the cable length using the mlnx_qos tool:

    # ofed_scripts/utils/mlnx_qos -i ens1f0 --cable_len 10
    DCBX mode: OS controlled
    Priority trust state: pcp
    Cable len: 10

    ...