HowTo Configure Trust State on Mellanox Adapters

Version 7

    This post explains how to set the trust state (either L2 PCP, or L3 DSCP) on Mellanox adapter cards while QoS is enabled on the network.

    Configuring trust mode to DSCP and enabling PFC allows running PFC based on the L3 DSCP field rather than the L2 PCP field. This will eliminate the need for VLAN in the Ethernet header.

    This feature is supported starting from MLNX_OFED v4.2 for ConnectX-4 adapters cards family and above.

     

    References

     

    Trust State

    Trust state has two values:

    • pcp - mapping pcp to priority. The default mapping is 1 to 1.
    • dscp - mapping dscp to priority. The default mapping is priority[dscp] = dscp>>3.

    The default trust state is "pcp".

     

    Configuration

    Enabling DSCP based flow control on DSCP 26:

    1. Use mlnx_qos  --trust=dscp to change the priority trust state from “pcp” to “dscp” on interface ens1f0.

    Only when the trust state is "dscp", dscp priority mapping is displayed.

    mlnx_qos -i ens1f0 --trust dscp
    DCBX mode: OS controlled
    Priority trust state: dscp
    dscp2prio mapping:
            prio:0 dscp:07,06,05,04,03,02,01,00,
            prio:1 dscp:15,14,13,12,11,10,09,08,
            prio:2 dscp:23,22,21,20,19,18,17,16,
            prio:3 dscp:31,30,29,28,27,26,25,24,
            prio:4 dscp:39,38,37,36,35,34,33,32,
            prio:5 dscp:47,46,45,44,43,42,41,40,
            prio:6 dscp:55,54,53,52,51,50,49,48,
            prio:7 dscp:63,62,61,60,59,58,57,56,
    Cable len: 10
    PFC configuration:
            priority    0   1   2   3   4   5   6   7
            enabled     1   0   1   0   0   0   1   1
    tc: 0 ratelimit: unlimited, tsa: vendor
             priority:  1
    tc: 1 ratelimit: unlimited, tsa: vendor
             priority:  0
    tc: 2 ratelimit: unlimited, tsa: vendor
             priority:  2
    tc: 3 ratelimit: unlimited, tsa: vendor
             priority:  3
    tc: 4 ratelimit: unlimited, tsa: vendor
             priority:  4
    tc: 5 ratelimit: unlimited, tsa: vendor
             priority:  5
    tc: 6 ratelimit: unlimited, tsa: vendor
             priority:  6
    tc: 7 ratelimit: unlimited, tsa: vendor
             priority:  7

     

     

    2. Since DSCP 26 (11010b) is mapped to priority 3 (11b), enable PFC for priority 3. The adapter will map priority 3 to the lossless buffer.

    # mlnx_qos -i ens1f0 --pfc 0,0,0,1,0,0,0,0

     

    3. Set the default RoCE version to v2 for RDMA CM.

    # cma_roce_mode -d mlx5_0 -p 1 -m 2

     

    4. To set DSCP for a RoCEv2 packet, use the following command:

    # cma_roce_tos –d mlx5_0 -t 104

     

    See HowTo Set Egress ToS/DSCP on RDMA-CM QPs  for more details.

     

    Verification

    Run option -T to set the type of service value to 104 (104 >> 2 is DSCP 26) in case you have not set the default ToS.

    Client

    # ib_send_bw -d mlx5_0 -x 3 -R -T 104

     

    Server

    # ib_send_bw -d mlx5_0 -x 3 -R -T 104 192.168.200.1