Linux RoCE Configuration in DSCP-Based QoS Mode

Version 16

    This post provides a configuration example of Resilient RoCE for Linux driver in DSCP-based QoS mode.

    For other RoCE Profile solutions, see Getting Started with RoCE Configuration.





    This solution involves a simple network setup and basic configuration on the adapter.


    Note: Some of the configuration steps below can either be done permanently or temporarily (can be kept for the next boot).

    For permanent configuration after running mlxconfig, a device reset (mlxfwreset) or host reboot is required.



    1. Set DSCP (L3) as trust mode for the NIC.

    # mlnx_qos -i <interface> --trust dscp

    For more information, see HowTo Configure Trust State on Mellanox Adapters.


    2. Enable DCQCN on priority 3 (used for RoCE traffic).

    Firmware configuration (non-volatile):

    # mlxconfig -d /dev/mst/<mst-device> -y s ROCE_CC_PRIO_MASK_P1=8 RPG_THRESHOLD_P1=1 DCE_TCP_G_P1=1019 ROCE_CC_PRIO_MASK_P2=8 RPG_THRESHOLD_P2=1 DCE_TCP_G_P2=1019


    Driver configuration (volatile):

    # echo 1 > /sys/class/net/<interface>/ecn/roce_np/enable/3

    # echo 1 > /sys/class/net/<interface>/ecn/roce_rp/enable/3

    # echo 1 > /sys/class/net/<interface>/ecn/roce_rp/rpg_threshold

    # echo 1019 > /sys/class/net/<interface>/ecn/roce_rp/dce_tcp_g


    2. Set the CNP priority to 48.

    Firmware configuration (non-volatile):

    # mlxconfig -d /dev/mst/<mst-device> -y s CNP_DSCP_P1=48 CNP_DSCP_P2=48


    Driver configuration (volatile):

    # echo 48 > /sys/class/net/<interface>/ecn/roce_np/cnp_dscp


    3. [Optional] Enable ECN for TCP traffic.

    # sysctl -w net.ipv4.tcp_ecn=1

    net.ipv4.tcp_ecn = 1

    Note: This command is nonpersistent.


    4. Set the RoCE mode to V2 for RDMA CM traffic.

    # cma_roce_mode -d mlx5_0 -p 1 -m 2

    For more details, see HowTo Set the Default RoCE Mode When Using RDMA CM.


    5. Set the default ToS to 106 (DSCP 26).

    # cma_roce_tos -d mlx5_0 -t 106

    For more information, see HowTo Set Egress ToS/DSCP on RDMA-CM QPs.


    For end to end configuration example and troubleshooting, see HowTo Configure Resilient RoCE (ECN) End-to-End Using ConnectX-4 and Spectrum (Trust L3).