RoCE Configuration for Linux Drivers without Setting QoS - OFED version 4.1

Version 21

    Note - this post is relevant for MLNX_OFED version 4.1. For versions 4.2 (specifically FW versions XX.21.1000) and above no configuration is required for this case.

    This post describes the configuration of resilient RoCE for Linux drivers without setting QoS.

    For other RoCE Profile solutions, see Getting Started with RoCE Configuration.






    This solution involves a simple network setup and basic configuration on the adapter.



    <interface> refers to parent interface (for example ens2f0)

    <mst-device> refers to MST device. (for example  /dev/mst/mt4115_pciconf0)

    <mlx-device> refers to mlx device (for example mlx5_0) 



    1. Enable DCQCN (ECN) on all priorities.

    Firmware Configuration (persistent):

    # mlxconfig -d /dev/mst/<mst-device> -y s ROCE_CC_PRIO_MASK_P1=255 RPG_THRESHOLD_P1=1 DCE_TCP_G_P1=1019 ROCE_CC_PRIO_MASK_P2=255 RPG_THRESHOLD_P2=1 DCE_TCP_G_P2=1019


    Driver Configuration (non-persistent):

    # echo 1 > /sys/class/net/<interface>/ecn/roce_np/enable/*

    # echo 1 > /sys/class/net/<interface>/ecn/roce_rp/enable/*

    # echo 1 > /sys/class/net/<interface>/ecn/roce_rp/rpg_threshold

    # echo 1019 > /sys/class/net/<interface>/ecn/roce_rp/dce_tcp_g

    For additional information about DC-QCN configuration, see HowTo Configure DCQCN (RoCE CC) for ConnectX-4 (Linux).


    2. Configure RDMA-CM (non-persistent) to be RoCEv2 by default.

    # cma_roce_mode -d <mlx_device> -p 1 -m 2     # Setting the RDMA Version to RoCEv2

    For more information, see HowTo Set the Default RoCE Mode When Using RDMA CM.


    3. If applicable, enable ECN for TCP traffic.

    # sysctl -w net.ipv4.tcp_ecn=1

    Note: To get mlx_dev (e.g. mlx5_0) from interface name, run:

    # ibdev2netdev | grep <interface> | cut -d' ' -f1