Recommended Network Configuration Examples for RoCE Deployment - new

Version 1

    This post provides recommended network configuration examples for deploying RoCE with Mellanox adapters.

    The below information is applicable for ConnectX-4 and above adapter cards, with MLNX_OFED version 4.2 or WinOF-2 version 1.80, including FW versions XX.21.1000 and above


    >> Learn  RDMA on the Mellanox Academy for free




    Network Profiles

    Review the network considerations section, select an appropriate column and follow the recipes in that columns


    Network Considerations
    Configuration modeLossy fabric
    Lossy fabric with QoSLossless fabric
    Recommended for

    Ease of deployment

    Minimal fabric configuration

    Larger scale with mixed traffic

    (TCP/UDP and RoCE)

    Uncompromised performance

    Storage deployment with heavy back-pressure from PCI

    Trust layer for QoS/PFCN/A

    L3 (DSCP)

    L2 (PCP)L3 (DSCP)L2 (PCP)
    VLAN requiredNoNoYesNoYes
    ECN (congestion control)YesYesYesYesYes
    PFC (flow control)NoNoNoYesYes
    Recommended configurations
    Configuration modeLossy fabricLossy fabric with QoSLossless fabric
    AdapterLinuxout of the boxlinklinklinklink
    Windowsout of the boxlinkN/AlinkN/A


    Additional switch configurations

    For optimal performance over RoCE transport we recommend using Mellanox end-to-end (Switch, Adapters and Cables).

    Mellanox Adapters and Switches are fully interoperable with 3rd party Network elements. Below are verified configuration for non-Mellanox Switches.



    Detailed Configuration Examples

    RDMA/RoCE Configuration for Linux (ConnectX-4/-5, mlx5 driver)

    Getting Started


    Extended Configuration examples


    Debugging and Troubleshooting


    RDMA/RoCE Configuration for Windows (ConnectX-4/5, WinOF-2 driver)


    RDMA/RoCE Configuration for Windows (ConnectX-3, WinOF driver)

    Getting Started

    Advance Considerations


    Other Switch Configurations

    Detailed configuration examples