HowTo Configure Trust Mode on Mellanox Adapters

Version 4

    This post shows how to set the trust mode (either L2 PCP, or L3 DSCP) on Mellanox adapters when QoS is used in the network.

    Having RoCEv2 enabled on the interface is useful. Setting the trust level to DSCP can eliminate the need for VLAN in the Ethernet header.

    Configuring trust mode to DSCP and enabling PFC allows you to run PFC based on the L3 DSCP field rather than the L2 PCP field.

    This is supported starting with MLNX_OFED 4.1 for ConnectX-4 adapters or newer.

     

    References

     

    Trust Mode

    There are two options for the trust parameter  /sys/class/net/eth1/qos/trust

    • pcp - the pcp field is mapped to priority. The default mapping is 1 to 1.
    • dscp - the DSCP field is mapped to priority. The default mapping is priority[dscp] = dscp>>3 (Taking the 3 most MSBs)

     

    The default priority of trust mode is "pcp".

     

    Note: This is applicable for RoCEv2 traffic.

    Configuration

    Example of enabling DSCP based flow control on DSCP 26:

    1. Set the priority trust mode from “pcp” to “dscp” on interface eth1.

    # cat /sys/class/net/eth1/qos/trust

    pcp

     

    # echo dscp > /sys/class/net/ens1f1/qos/trust

     

    # cat /sys/class/net/ens1f1/qos/trust

    dscp

     

    another option is to use mlnx_qos  --trust=dscp.

     

    #  mlnx_qos -i eth2 --trust=dscp

    Priority trust mode: dscp

    PFC configuration:

      priority    0   1   2   3   4   5   6   7

      enabled     0   0   0   0   0   0   0   0  

     

    tc: 0 ratelimit: unlimited, tsa: ets, bw: 10%

      priority:  1

    tc: 1 ratelimit: unlimited, tsa: ets, bw: 10%

      priority:  0

    tc: 2 ratelimit: unlimited, tsa: ets, bw: 10%

      priority:  2

    tc: 3 ratelimit: unlimited, tsa: ets, bw: 10%

      priority:  3

    tc: 4 ratelimit: unlimited, tsa: ets, bw: 10%

      priority:  4

    tc: 5 ratelimit: unlimited, tsa: ets, bw: 10%

      priority:  5

    tc: 6 ratelimit: unlimited, tsa: strict

      priority:  6

    tc: 7 ratelimit: unlimited, tsa: ets, bw: 40%

      priority:  7

     

    # cat /sys/class/net/eth2/qos/trust

    dscp

     

    2. Because DSCP 26 (11010b) is mapped to priority 3 (11b), enable PFC for priority 3. The adapter will map priority 3 to the lossless buffer.

    # mlnx_qos -i eth1 --pfc 0,0,0,1,0,0,0,0

     

    3. Set the default RoCE version to v2 for RDMA CM.

    # cma_roce_mode -d mlx5_0 -p 1 -m 2

     

    4. To set DSCP for a RoCEv2 packet, use the following command:

    # cma_roce_tos –d mlx5_0 -t 104

     

    See HowTo Set Egress ToS/DSCP on RDMA-CM QPs  for more details.

     

    Verification

    Use option -T to set the type of service value to 104 (104 >> 2 is DSCP 26) if you haven't set the default ToS.

    Client

    # ib_send_bw -d mlx5_0 -x 3 -R -T 104

     

    Server

    # ib_send_bw -d mlx5_0 -x 3 -R -T 104 192.168.200.1