Resilient RoCE Configuration for Linux Drivers without Setting QoS

Version 17

    This post describes the configuration of resilient RoCE for Linux drivers without setting QoS.

    For other RoCE Profile solutions, see Getting Started with RoCE Configuration.

     

    References

     

     

    Overview

    This solution involves a simple network setup and basic configuration on the adapter.

     

    Notations

    <interface> refers to parent interface (for example ens2f0)

    <mst-device> refers to MST device. (for example  /dev/mst/mt4115_pciconf0)

    <mlx-device> refers to mlx device (for example mlx5_0) 

     

    Configuration

    1. Enable DCQCN (ECN) on all priorities.

    Firmware Configuration (persistent):

    # mlxconfig -d /dev/mst/<mst-device> -y s ROCE_CC_PRIO_MASK_P1=255 ROCE_CC_PRIO_MASK_P2=255

    OR

    Driver Configuration (non-persistent):

    # echo 1 > /sys/class/net/<interface>/ecn/roce_np/enable/*

    # echo 1 > /sys/class/net/<interface>/ecn/roce_rp/enable/*

    For additional information about DC-QCN configuration, see HowTo Configure DCQCN (RoCE CC) for ConnectX-4 (Linux).

     

    2. Configure RDMA-CM (non-persistent) to be RoCEv2 by default.

    # cma_roce_mode -d <mlx_device> -p 1 -m 2     # Setting the RDMA Version to RoCEv2

    For more information, see HowTo Set the Default RoCE Mode When Using RDMA CM.

     

    3. If applicable, enable ECN for TCP traffic.

    # sysctl -w net.ipv4.tcp_ecn=1

    Note: To get mlx_dev (e.g. mlx5_0) from interface name, run:

    # ibdev2netdev | grep <interface> | cut -d' ' -f1