Recommended Network Configuration Examples for RoCE Deployment

Version 67

    This post provides recommended network configuration examples for deploying RoCE with Mellanox adapters.

    The below information is applicable for ConnectX-4 and above adapter cards, with MLNX_OFED version 4.2 or WinOF-2 version 1.80, including FW versions XX.21.1000 and above

     

    >> Learn  RDMA on the Mellanox Academy for free

     

    References

     

    Network Profiles

     

    Configuration Mode

    Trust

    VLAN Required

    ECN

    PFC

    RDMA Priority

    Adapter

    Switch

     

     

     

     

     

     

    Linux

    Windows

    MLNX-OS / Onyx

    Cumulus Linux

    Lossy Fabric

    N/A

    No

    Yes

    No

    Any

    preconfigured

    preconfigured

    link

    soon

    Lossy Fabric with QoS

    Layer 3

    No

    Yes

    No

    DSCP = 26

    link

    link

    link

    soon

    Layer 2

    Yes

    Yes

    No

    PCP = 3

    link

    N/A

    link

    soon

    Lossless fabric

    Layer 3

    No

    Yes

    Yes

    DSCP = 26

    link

    link

    link

    link

    Layer 2

    Yes

    Yes

    Yes

    PCP = 3

    link

    N/A

    link

    soon

     

    To learn more about the network characteristics and profile selection, see: Understanding QoS Configuration for RoCE

     

    Other Vendors Tested Configurations

     

    Detailed Configuration Examples

    RDMA/RoCE Configuration for Linux (ConnectX-4/-5, mlx5 Driver)

    Getting Started

     

    Advanced Considerations

     

    Debugging and Troubleshooting

     

    RDMA/RoCE Configuration for Windows (ConnectX-4/5, WinOF-2 Driver)

     

    RDMA/RoCE Configuration for Windows (ConnectX-3, WinOF Driver)

    Getting Started

    Advanced Considerations

     

    Other Vendors Switch Configurations

    Detailed Configuration Examples