RoCE Configuration for Mellanox Adapters (Profile 5)

Version 12

    The following post shows a configuration example for Linux to enable L3 priority (DSCP) based Lossless RoCE traffic, when using trust L3 in the switch configuration. This method described as Profile 5.

    For other RoCE Profile solutions see Getting Started with RoCE Configuration . The configuration example is using DSCP 26 (ToS 105) for RoCE traffic and DSCP 48 for CNP traffic.

     

    References

     

    Overview

    This solutions has the simple network assumption and basic configuration on the adapter.

     

    Some of the configuration below can be permanent (non-volatile) or temporary (volatile).

    •      Permanent configuration -  device reset (mlxfwreset) or host reboot is required.
    •      Temporary configuration - affects immediately but erased in a next boot.

     

     

    Configuration

    1. Enable DCQCN on priority 3 (RoCE traffic).

     

    Option 1: non-volatile

    # mlxconfig -d /dev/mst/<mst-device> -y s ROCE_CC_PRIO_MASK_P1=8 ROCE_CC_PRIO_MASK_P2=8

     

    Option 2: volatile

    # echo 1 > /sys/class/net/<interface>/ecn/roce_np/enable/3

    # echo 1 > /sys/class/net/<interface>/ecn/roce_rp/enable/3

     

    2. Set trust DSCP mode

    # echo dscp > /sys/class/net/<interface>/qos/trust

     

    For more info, see HowTo Configure Trust Mode on Mellanox Adapters.

     

    3. Set CNP DSCP to 48

    Option 1: non-volatile

    # mlxconfig -d /dev/mst/<mst-device> -y s CNP_DSCP_P1=48 CNP_DSCP_P2=48

     

    Option 2: volatile

    # echo 48 > /sys/class/net/<interface>/ecn/roce_np/cnp_dscp

     

    4. (Optional) Enable ECN for TCP traffic:

    # sysctl -w net.ipv4.tcp_ecn=1

    net.ipv4.tcp_ecn = 1

     

    Note: This command is not persistent.

     

    4. Set the RoCE mode to V2 using RDMA-CM

    # cma_roce_mode -d mlx5_0 -p 1 -m 2

    For more details, see HowTo Set the Default RoCE Mode When Using RDMA CM.

     

    5. Set RoCE DSCP tp 26 (ToS 105) using RDMA-CM

    # cma_roce_tos -d mlx5_0 -t 105

    For more information, see HowTo Set Egress ToS/DSCP on RDMA-CM QPs.

     

    6.  Activate PFC on priority 3.

    Option 1: non-volatile using mlnx_qos tool

    # mlnx_qos -i <interface> --pfc 0,0,0,1,0,0,0,0

     

    See also, HowTo Configure PFC on ConnectX-4

     

    Option 2: non-volatile using LLDP DCBX. Getting the configuration from the switch

    # mlxconfig -d /dev/mst/mt4115_pciconf0 -y s LLDP_NB_DCBX_P1=TRUE LLDP_NB_TX_MODE_P1=2 LLDP_NB_RX_MODE_P1=2 LLDP_NB_DCBX_P2=TRUE LLDP_NB_TX_MODE_P2=2 LLDP_NB_RX_MODE_P2=2

    See also: HowTo Auto-Config PFC and ETS on ConnectX-4 via LLDP DCBX

    Note: requires LLDP service enabled in the switch

     

    Option 3: volatile using LLDP DCBX. Getting the configuration from the switch.

    # service lldpad start

    # lldptool -T -i <interface_name> -V PFC -c willing=yes

    See also: HowTo Auto-Config PFC and ETS on ConnectX-4 via LLDP DCBX

    Note: requires LLDP service enabled in the switch