Recommended Network Configuration Examples for RoCE Deployment - new

Version 3

    This post provides recommended network configuration examples for deploying RoCE with Mellanox ConnectX and BlueField based adapters and Spectrum switches.

    The below information is applicable for:

    ConnectX-4/4LX/5 and BlueField adapter cards, with MLNX_OFED version 4.2 or WinOF-2 version 1.90, including FW versions XX.22.1000 and above

    Spectrum switches with Onyx version 3.6.5000 and above


    >> Learn  RDMA on the Mellanox Academy for free


    Recipes for RoCE fabrics

    Review the Fabric Attributes section, select an appropriate column and follow the recipes in that columns


    Fabric Attributes
    Fabric ConfigurationLossy fabric
    Lossy fabric with QoSLossless fabric
    Recommended for

    Ease of deployment

    Minimal fabric configuration

    Large scale with mixed traffic

    (TCP/UDP and RoCE)

    Uncompromised performance over large scale

    Storage deployment with heavy back-pressure from PCI**

    Trust layer for QoS/PFC*N/A

    L3 (DSCP)

    L2 (PCP)L3 (DSCP)L2 (PCP)
    VLAN requiredNoNoYesNoYes
    ECN (congestion control)YesYesYesYesYes
    PFC (flow control)NoNoNoYesYes
    Recommended Recipes
    Fabric ConfigurationLossy fabricLossy fabric with QoSLossless fabric


    OS and required versions

    LinuxMLNX_OFED 4.2out of the boxlinklinklinklink
    WindowsWinOF 1.90out of the boxlinkN/AlinkN/A


    OS and required versions



    * For more information see: Network_Flow_Classification

    ** PCI backpressure can happen when working with multiple storage devices or when oversubscribing ports to PCI (2x100G links on 16-lane PCI gen3)


    Non-Mellanox switch configurations

    For optimal performance over RoCE transport we recommend using Mellanox end-to-end (Spectrum switches, ConnectX and BlueField Adapters and LinkX Cables).

    Mellanox Adapters and Switches are fully interoperable with 3rd party network elements. Below are verified configuration for non-Mellanox Switches.





    Detailed Configuration Examples

    RDMA/RoCE Configuration for Linux (ConnectX-4/-5, mlx5 driver)

    Getting Started


    Extended Configuration examples


    Debugging and Troubleshooting


    RDMA/RoCE Configuration for Windows (ConnectX-4/5, WinOF-2 driver)