Linux RoCE Configuration in DSCP-Based QoS Mode

Version 19

    This post provides a configuration example of Resilient RoCE for Linux driver in DSCP-based QoS mode.

    For other RoCE Profile solutions, see Getting Started with RoCE Configuration.

     

    References

     

    Overview

    This solution involves a simple network setup and basic configuration on the adapter.

     

    Note: Some of the configuration steps below can either be done permanently or temporarily (can be kept for the next boot).

    For permanent configuration after running mlxconfig, a device reset (mlxfwreset) or host reboot is required.

     

    Configuration

    1. Set DSCP (L3) as trust mode for the NIC.

    # mlnx_qos -i <interface> --trust dscp

    For more information, see HowTo Configure Trust State on Mellanox Adapters.

     

    2. [Optional] Enable ECN for TCP traffic.

    # sysctl -w net.ipv4.tcp_ecn=1

    net.ipv4.tcp_ecn = 1

    Note: This command is nonpersistent.

     

    3. Set the default ToS to 106 (DSCP 26).

    # cma_roce_tos -d mlx5_0 -t 106

    For more information, see HowTo Set Egress ToS/DSCP on RDMA-CM QPs.

     

    For end to end configuration example and troubleshooting, see HowTo Configure Resilient RoCE (ECN) End-to-End Using ConnectX-4 and Spectrum (Trust L3).