Recommended Network Configuration Examples for RoCE Deployment

Version 57

    This post provides recommended network configuration examples for deploying RoCE with Mellanox adapters.

    The below information is applicable for ConnectX-4 and above adapter cards, with MLNX_OFED version 4.2 or WinOF-2 version 1.80, including FW versions XX.21.1000 and above

     

    >> Learn  RDMA on the Mellanox Academy for free

     

    References

     

    Network Profiles

     

    Configuration Mode

    Trust

    VLAN Required

    ECN

    PFC

    RDMA Priority

    Adapter

    Switch

     

     

     

     

     

     

    Linux

    Windows

    MLNX_OS

    Cumulus
    Lossy fabric

    N/A

    No

    Yes

    No

    Any

    preconfigured

    preconfigured

    link

    soon

    Lossy fabric with QoS

    Layer 3

    No

    Yes

    No

    DSCP = 26

    link

    link

    link

    soon

    Layer 2

    Yes

    Yes

    No

    PCP = 3

    link

    N/A

    link

    soon

    Lossless fabric

    Layer 3

    No

    Yes

    Yes

    DSCP = 26

    link

    link

    link

    link

    Layer 2

    Yes

    Yes

    Yes

    PCP = 3

    link

    N/A

    link

    soon

     

    To learn more about the network characteristics and profile selection, see: Understanding QoS Configuration for RoCE

     

    Detailed Configuration Examples

    RDMA/RoCE Configuration for Linux (ConnectX-4/-5, mlx5 driver)

    Getting Started

     

    Advanced Considerations

     

    Debugging and Troubleshooting

     

    RDMA/RoCE Configuration for Windows (ConnectX-4/5, WinOF-2 driver)

     

    RDMA/RoCE Configuration for Windows (ConnectX-3, WinOF driver)

    Getting Started

    Advance Considerations

     

    Other vendors Switch Configurations

     

    Detailed configuration examples