Introduction to Resilient RoCE - FAQ

Version 4

    This post is an introduction to Resilient RoCE for ConnectX-4.






    What is Resilient RoCE?

    Resilient RoCE is the ability to send RoCE traffic over a lossy network (a network without flow control enabled), without the need to enable flow control on the network.


    Is this a different version of RoCE?

    No, there is no change or addition to the RoCE spec. It assumes RoCEv2 implementation. This is not supported for RoCEv1.


    Which adapter cards support this?

    Resilient RoCE is supported on ConnectX-4/Lx adapters and onward on the hardware level.

    For ConnectX-3 Pro adapters, we recommend to always have flow control configured (and optionally ECN) when RoCE is used in the network.


    What types of congestion control mechanisms should I enable on my network?

    It is recommended to enable RoCE congestion control (ECN) as well as PFC. However, PFC or general flow control is not a must requirement. In some cases, network administrators prefer not to handle PFC configuration due to the possible complexity of the configuration.


    What is RoCE congestion control?

    Refer to the RoCEv2 Congestion Management community post.


    What are the prerequisites from the switches on the network?

    The network's switches should support the ability to:

    • Mark ECN bits on the IP header in congestion scenarios
    • Map priorities to traffic classes for QoS


    What are the ECN bits on that IP header?

    The IP header has two bits for ECN with the following options:

    00 - no ECN capable

    01 - ECN capable

    10 - ECN capable

    11 - congestion encountered

    When the network is congested, the switches change those two bits from 01 or 10 (there is no difference) to 11, assuming the switches have ECN enabled.


    If only RoCE congestion control (ECN) is used without flow control, does that assure a lossless network?

    No, unlike flow control (or PFC), drops may occur in case of congestion.


    How do dropped packets affect the RoCE/RDMA connection?

    Dropped packets cause disruption to the traffic flow. Due to the semantics of InfiniBand, the responder receives only packets in the right sequence.

    How is packet loss handled?

    Upon packet drop, a NACK (not ACK) control packet with the specific PSN (packet sequence number) is sent to the sender in order for it to re-transmit the packet.


    Where is Resilient RoCE supported?

    • Resilient RoCE is supported for Linux when using MLMX_OFED (2016 May release is beta)
    • ECN is supported on Mellanox Spectrum™ based switches
    • Auto-configuration is supported for ConnectX-4 and Mellanox Spectrum based switches using Mellanox NEO

    How do I configure ECN on Mellanox Spectrum™ based switches?

    Refer to HowTo Configure ECN on Mellanox Ethernet Switches (Spectrum).


    How do I configure ECN on ConnectX-4 in a Linux environment?

    Refer to HowTo Configure ECN for ConnectX-4 (Linux).