RoCE Configuration for Mellanox Adapters (PCP-Based Resilient)

Version 13

    This post provides a Linux configuration example for enabling L2 priority (Priority Code Point [PCP]) based lossy RoCE traffic, when using Trust L2 in the switch configuration. This method is described as Profile 2.

    For other RoCE Profile solutions, see Getting Started with RoCE Configuration .





    This solution involves a simple network setup and basic configuration on the adapter.



    <interface> refers to parent interface (for example ens2f0)

    <vlan-interface> refers to vlan interface (for example ens2f0.100)

    <mst-device> refers to MST device. (for example  /dev/mst/mt4115_pciconf0)

    <mlx-device> refers to mlx device (for example mlx5_0) 


    Note: Some of the configuration steps below can either be done permanently or temporarily (can be kept for the next boot).



    1. Enable DCQCN on priority 3 (used for RoCE traffic).

    Firmware configuration (non-volatile):

    # mlxconfig -d /dev/mst/<mst-device> -y s ROCE_CC_PRIO_MASK_P1=8 ROCE_CC_PRIO_MASK_P2=8


    Driver configuration (volatile):

    # echo 1 > /sys/class/net/<interface>/ecn/roce_np/enable/3

    # echo 1 > /sys/class/net/<interface>/ecn/roce_rp/enable/3


    2. Set the CNP priority to 6.

    Firmware configuration (non-volatile):

    # mlxconfig -d /dev/mst/<mst-device> -y s CNP_802P_PRIO_P1=6 CNP_802P_PRIO_P2=6


    Driver configuration (volatile):

    # echo 6 > /sys/class/net/<interface_name>/ecn/roce_np/cnp_802p_prio


    3. Configure RDMA-CM (volatile).

    # cma_roce_mode -d <mlx-device> -p 1 -m 2        # Set the RDMA Version to RoCEv2

    # cma_roce_tos -d <mlx_dev> -t 105               # Setting the TOS for RDMA-CM to TOS=105, mapped to sk_prio=2

    # vconfig set_egress_map <vlan-interface> 2 3    # Map sk_prio=2 to SL=3


    4. [Optional] Enable ECN on TCP traffic and set egress priority for the TCP traffic (volatile).

    # sysctl -w net.ipv4.tcp_ecn=1

    # vconfig set_egress_map <vlan-interface> <sk_prio_number> <priority>


    For end to end configuration example and troubleshooting, see HowTo Configure Resilient RoCE (ECN) End-to-End Using ConnectX-4 and Spectrum (Trust L2).