HowTo Configure Trust state on Mellanox Adapters

Version 5

    This post shows how to set the trust mode (either L2 PCP, or L3 DSCP) on Mellanox adapters when QoS is used in the network.

    Having RoCEv2 enabled on the interface is useful. Setting the trust level to DSCP can eliminate the need for VLAN in the Ethernet header.

    Configuring trust mode to DSCP and enabling PFC allows you to run PFC based on the L3 DSCP field rather than the L2 PCP field.

    This is supported starting with MLNX_OFED 4.2 for ConnectX-4 adapters or newer.

     

    References

     

    Trust Mode

    There are two options for the trust state:

    • pcp - the pcp field is mapped to priority. The default mapping is 1 to 1.
    • dscp - the DSCP field is mapped to priority. The default mapping is priority[dscp] = dscp>>3.

     

    The default priority of trust mode is "pcp".

     

    Configuration

    Example of enabling DSCP based flow control on DSCP 26:

    1. Use mlnx_qos  --trust=dscp to set the priority trust mode from “pcp” to “dscp” on interface ens1f0. The dscp to priority mapping are displayed when the trust state is dscp.

    mlnx_qos -i ens1f0 --trust dscp
    DCBX mode: OS controlled
    Priority trust state: dscp
    dscp2prio mapping:
            prio:0 dscp:07,06,05,04,03,02,01,00,
            prio:1 dscp:15,14,13,12,11,10,09,08,
            prio:2 dscp:23,22,21,20,19,18,17,16,
            prio:3 dscp:31,30,29,28,27,26,25,24,
            prio:4 dscp:39,38,37,36,35,34,33,32,
            prio:5 dscp:47,46,45,44,43,42,41,40,
            prio:6 dscp:55,54,53,52,51,50,49,48,
            prio:7 dscp:63,62,61,60,59,58,57,56,
    Cable len: 10
    PFC configuration:
            priority    0   1   2   3   4   5   6   7
            enabled     1   0   1   0   0   0   1   1
    tc: 0 ratelimit: unlimited, tsa: vendor
             priority:  1
    tc: 1 ratelimit: unlimited, tsa: vendor
             priority:  0
    tc: 2 ratelimit: unlimited, tsa: vendor
             priority:  2
    tc: 3 ratelimit: unlimited, tsa: vendor
             priority:  3
    tc: 4 ratelimit: unlimited, tsa: vendor
             priority:  4
    tc: 5 ratelimit: unlimited, tsa: vendor
             priority:  5
    tc: 6 ratelimit: unlimited, tsa: vendor
             priority:  6
    tc: 7 ratelimit: unlimited, tsa: vendor
             priority:  7

     

     

    2. Because DSCP 26 (11010b) is mapped to priority 3 (11b), enable PFC for priority 3. The adapter will map priority 3 to the lossless buffer.

    # mlnx_qos -i ens1f0 --pfc 0,0,0,1,0,0,0,0

     

    3. Set the default RoCE version to v2 for RDMA CM.

    # cma_roce_mode -d mlx5_0 -p 1 -m 2

     

    4. To set DSCP for a RoCEv2 packet, use the following command:

    # cma_roce_tos –d mlx5_0 -t 104

     

    See HowTo Set Egress ToS/DSCP on RDMA-CM QPs  for more details.

     

    Verification

    Use option -T to set the type of service value to 104 (104 >> 2 is DSCP 26) if you haven't set the default ToS.

    Client

    # ib_send_bw -d mlx5_0 -x 3 -R -T 104

     

    Server

    # ib_send_bw -d mlx5_0 -x 3 -R -T 104 192.168.200.1