RoCE Configuration for Linux Drivers without Setting QoS

Version 20

    This post describes the configuration of resilient RoCE for Linux drivers without setting QoS.

    For other RoCE Profile solutions, see Getting Started with RoCE Configuration.

     

    References

     

     

    Overview

    This solution involves a simple network setup and basic configuration on the adapter.

     

    Notations

    <interface> refers to parent interface (for example ens2f0)

    <mst-device> refers to MST device. (for example  /dev/mst/mt4115_pciconf0)

    <mlx-device> refers to mlx device (for example mlx5_0) 

     

    Configuration

    1. Enable DCQCN (ECN) on all priorities.

    Firmware Configuration (persistent):

    # mlxconfig -d /dev/mst/<mst-device> -y s ROCE_CC_PRIO_MASK_P1=255 RPG_THRESHOLD_P1=1 DCE_TCP_G_P1=1019 ROCE_CC_PRIO_MASK_P2=255 RPG_THRESHOLD_P2=1 DCE_TCP_G_P2=1019

    OR

    Driver Configuration (non-persistent):

    # echo 1 > /sys/class/net/<interface>/ecn/roce_np/enable/*

    # echo 1 > /sys/class/net/<interface>/ecn/roce_rp/enable/*

    # echo 1 > /sys/class/net/<interface>/ecn/roce_rp/rpg_threshold

    # echo 1019 > /sys/class/net/<interface>/ecn/roce_rp/dce_tcp_g

    For additional information about DC-QCN configuration, see HowTo Configure DCQCN (RoCE CC) for ConnectX-4 (Linux).

     

    2. Configure RDMA-CM (non-persistent) to be RoCEv2 by default.

    # cma_roce_mode -d <mlx_device> -p 1 -m 2     # Setting the RDMA Version to RoCEv2

    For more information, see HowTo Set the Default RoCE Mode When Using RDMA CM.

     

    3. If applicable, enable ECN for TCP traffic.

    # sysctl -w net.ipv4.tcp_ecn=1

    Note: To get mlx_dev (e.g. mlx5_0) from interface name, run:

    # ibdev2netdev | grep <interface> | cut -d' ' -f1