Recommended Network Configuration Examples for RoCE Deployment

Version 74

    This post provides recommended network configuration examples for deploying RoCE with Mellanox ConnectX and BlueField based adapters and Spectrum switches.

    The below information is applicable for:

    ConnectX-4/4LX/5 and BlueField adapter cards, with MLNX_OFED version 4.2 or WinOF-2 version 1.90, including FW versions XX.22.1000 and above

    Spectrum switches with Onyx version 3.6.5000 and above

     

    >> Learn  RDMA on the Mellanox Academy for free

     

    Table of contents

    Recipes for RoCE fabrics

    Review the Fabric Attributes section, select an appropriate column and follow the recipes in that columns

     

    Fabric Attributes
    Fabric ConfigurationLossy fabric
    Lossy fabric with QoSLossless fabric
    Recommended for

    Ease of deployment

    Minimal fabric configuration

    Large scale with mixed traffic

    (TCP/UDP and RoCE)

    Uncompromised performance over large scale

    Storage deployment with heavy back-pressure from PCI*

    Trust layer for QoS/PFC**N/A

    L3 (DSCP)

    L3 (DSCP)
    VLAN requiredNoNoNo
    ECN (congestion control)YesYesYes
    PFC (flow control)NoNoYes
    Recommended Recipes
    Fabric ConfigurationLossy fabricLossy fabric with QoSLossless fabric

    Adapter

    OS and required versions

    LinuxMLNX_OFED 4.2out of the boxlinklink
    WindowsWinOF 2.0out of the boxlinklink

    Switch

    OS and required versions

    Onyx3.6.5000linklinklink
    Cumulus3.4soonsoon
    link

     

    * PCI backpressure can occur when working with multiple storage devices or when oversubscribing ports to PCI (2x100G links on 16-lane PCI gen3)

    ** For more information see: Network_Flow_Classification

     

    Note on L2 based QoS/PFC

    For simplicity, this page points to L3 based configurations. L3 based configurations can run on top of any network (including with/without VLANs) and is routable.

    As a reference, below are L2 based lossless configurations for OFED 4.3 and Onyx 3.6.5000:

    RoCE Configuration on Mellanox Adapters (PCP-Based Lossless Traffic)

    Lossless RoCE Configuration for Onyx Switches in PCP-Based QoS Mode

     

    Non-Mellanox switch configurations

    For optimal performance over RoCE transport we recommend using Mellanox end-to-end (Spectrum switches, ConnectX and BlueField Adapters and LinkX Cables).

    Mellanox Adapters and Switches are fully interoperable with 3rd party network elements. Below are verified configuration for non-Mellanox Switches.

     

    Related guides

     

    Debugging and troubleshooting

     

    Understanding RoCE