This post is an introduction to Resilient RoCE for ConnectX-4.
- What is Resilient RoCE?
- Is this a different version of RoCE?
- Which adapter cards support this?
- What types of congestion control mechanisms should I enable on my network?
- What is RoCE congestion control?
- What are the prerequisites from the switches on the network?
- What are the ECN bits on that IP header?
- If only RoCE congestion control (ECN) is used without flow control, does that assure a lossless network?
- How do dropped packets affect the RoCE/RDMA connection?
- How is packet loss handled?
- Where is Resilient RoCE supported?
- How do I configure ECN on Mellanox Spectrum™ based switches?
- How do I configure ECN on ConnectX-4 in a Linux environment?
- RDMA/RoCE Solutions
- RoCEv2 Congestion Management
- HowTo Configure RoCE Congestion Control for Windows 2012
- RoCEv2 CNP Packet Format Example
- Download Wireshark with RoCEv2 Support
- HowTo Configure CoS and DSCP for SMB in Windows 2012
- QoS Configuration examples for Cisco Nexus 5600
- Resilient RoCE Relaxes RDMA Requirements | Mellanox Technologies Blog Mellanox Technologies Blog
What is Resilient RoCE?
Resilient RoCE is the ability to send RoCE traffic over a lossy network (a network without flow control enabled), without the need to enable flow control on the network.
Is this a different version of RoCE?
No, there is no change or addition to the RoCE spec. It assumes RoCEv2 implementation. This is not supported for RoCEv1.
Which adapter cards support this?
Resilient RoCE is supported on ConnectX-4/Lx adapters and onward on the hardware level.
For ConnectX-3 Pro adapters, we recommend to always have flow control configured (and optionally ECN) when RoCE is used in the network.
What types of congestion control mechanisms should I enable on my network?
It is recommended to enable RoCE congestion control (ECN) as well as PFC. However, PFC or general flow control is not a must requirement. In some cases, network administrators prefer not to handle PFC configuration due to the possible complexity of the configuration.
What is RoCE congestion control?
Refer to the RoCEv2 Congestion Management community post.
What are the prerequisites from the switches on the network?
The network's switches should support the ability to:
- Mark ECN bits on the IP header in congestion scenarios
- Map priorities to traffic classes for QoS
What are the ECN bits on that IP header?
The IP header has two bits for ECN with the following options:
00 - no ECN capable
01 - ECN capable
10 - ECN capable
11 - congestion encountered
When the network is congested, the switches change those two bits from 01 or 10 (there is no difference) to 11, assuming the switches have ECN enabled.
If only RoCE congestion control (ECN) is used without flow control, does that assure a lossless network?
No, unlike flow control (or PFC), drops may occur in case of congestion.
How do dropped packets affect the RoCE/RDMA connection?
Dropped packets cause disruption to the traffic flow. Due to the semantics of InfiniBand, the responder receives only packets in the right sequence.
How is packet loss handled?
Upon packet drop, a NACK (not ACK) control packet with the specific PSN (packet sequence number) is sent to the sender in order for it to re-transmit the packet.
Where is Resilient RoCE supported?
- Resilient RoCE is supported for Linux when using MLMX_OFED (2016 May release is beta)
- ECN is supported on Mellanox Spectrum™ based switches
- Auto-configuration is supported for ConnectX-4 and Mellanox Spectrum based switches using Mellanox NEO
How do I configure ECN on Mellanox Spectrum™ based switches?