Recommended Network Configuration Examples for RoCE Deployment - new

Version 3

    This post provides recommended network configuration examples for deploying RoCE with Mellanox ConnectX and BlueField based adapters and Spectrum switches.

    The below information is applicable for:

    ConnectX-4/4LX/5 and BlueField adapter cards, with MLNX_OFED version 4.2 or WinOF-2 version 1.90, including FW versions XX.22.1000 and above

    Spectrum switches with Onyx version 3.6.5000 and above

     

    >> Learn  RDMA on the Mellanox Academy for free

     

    Recipes for RoCE fabrics

    Review the Fabric Attributes section, select an appropriate column and follow the recipes in that columns

     

    Fabric Attributes
    Fabric ConfigurationLossy fabric
    Lossy fabric with QoSLossless fabric
    Recommended for

    Ease of deployment

    Minimal fabric configuration

    Large scale with mixed traffic

    (TCP/UDP and RoCE)

    Uncompromised performance over large scale

    Storage deployment with heavy back-pressure from PCI**

    Trust layer for QoS/PFC*N/A

    L3 (DSCP)

    L2 (PCP)L3 (DSCP)L2 (PCP)
    VLAN requiredNoNoYesNoYes
    ECN (congestion control)YesYesYesYesYes
    PFC (flow control)NoNoNoYesYes
    Recommended Recipes
    Fabric ConfigurationLossy fabricLossy fabric with QoSLossless fabric

    Adapter

    OS and required versions

    LinuxMLNX_OFED 4.2out of the boxlinklinklinklink
    WindowsWinOF 1.90out of the boxlinkN/AlinkN/A

    Switch

    OS and required versions

    Onyx3.6.5000linklinklinklinklink
    Cumulus3.4soonsoonsoonlinksoon

     

    * For more information see: Network_Flow_Classification

    ** PCI backpressure can happen when working with multiple storage devices or when oversubscribing ports to PCI (2x100G links on 16-lane PCI gen3)

     

    Non-Mellanox switch configurations

    For optimal performance over RoCE transport we recommend using Mellanox end-to-end (Spectrum switches, ConnectX and BlueField Adapters and LinkX Cables).

    Mellanox Adapters and Switches are fully interoperable with 3rd party network elements. Below are verified configuration for non-Mellanox Switches.

     

     

    References

     

    Detailed Configuration Examples

    RDMA/RoCE Configuration for Linux (ConnectX-4/-5, mlx5 driver)

    Getting Started

     

    Extended Configuration examples

     

    Debugging and Troubleshooting

     

    RDMA/RoCE Configuration for Windows (ConnectX-4/5, WinOF-2 driver)