Recommended Network Configuration Examples for RoCE Deployment - new

Version 1

    This post provides recommended network configuration examples for deploying RoCE with Mellanox adapters.

    The below information is applicable for ConnectX-4 and above adapter cards, with MLNX_OFED version 4.2 or WinOF-2 version 1.80, including FW versions XX.21.1000 and above

     

    >> Learn  RDMA on the Mellanox Academy for free

     

    References

     

    Network Profiles

    Review the network considerations section, select an appropriate column and follow the recipes in that columns

     

    Network Considerations
    Configuration modeLossy fabric
    Lossy fabric with QoSLossless fabric
    Recommended for

    Ease of deployment

    Minimal fabric configuration

    Larger scale with mixed traffic

    (TCP/UDP and RoCE)

    Uncompromised performance

    Storage deployment with heavy back-pressure from PCI

    Trust layer for QoS/PFCN/A

    L3 (DSCP)

    L2 (PCP)L3 (DSCP)L2 (PCP)
    VLAN requiredNoNoYesNoYes
    ECN (congestion control)YesYesYesYesYes
    PFC (flow control)NoNoNoYesYes
    Recommended configurations
    Configuration modeLossy fabricLossy fabric with QoSLossless fabric
    AdapterLinuxout of the boxlinklinklinklink
    Windowsout of the boxlinkN/AlinkN/A
    SwitchMLNX-OSlinklinklinklinklink
    Cumulussoonsoonsoonlinksoon

     

    Additional switch configurations

    For optimal performance over RoCE transport we recommend using Mellanox end-to-end (Switch, Adapters and Cables).

    Mellanox Adapters and Switches are fully interoperable with 3rd party Network elements. Below are verified configuration for non-Mellanox Switches.

     

     

    Detailed Configuration Examples

    RDMA/RoCE Configuration for Linux (ConnectX-4/-5, mlx5 driver)

    Getting Started

     

    Extended Configuration examples

     

    Debugging and Troubleshooting

     

    RDMA/RoCE Configuration for Windows (ConnectX-4/5, WinOF-2 driver)

     

    RDMA/RoCE Configuration for Windows (ConnectX-3, WinOF driver)

    Getting Started

    Advance Considerations

     

    Other Switch Configurations

    Detailed configuration examples