This post provides a configuration example of RoCE over lossy networks for MLNX-OS switches in PCP-based QoS mode.
- Mellanox adapters and switches support DSCP based QoS and flow control, which is easier and simpler to configure and doesn't require VLANS, QoS is maintained across routers.
- This post assumes MLNX-OS version 3.6.5000 and above. For manual buffer configuration and older versions see: RoCE Configuration for MLNX-OS Switches in PCP-Based QoS Mode (advanced mode)
- For other RoCE Profile solutions, see Recommended Network Configuration Examples for RoCE Deployment
- Recommended Network Configuration Examples for RoCE Deployment
- Ethernet Switch Solutions
- HowTo Configure ECN on Mellanox Ethernet Switches (Spectrum)
- RoCE Configuration for Mellanox Adapters (PCP-Based)
This solutions offers the following network setup:
- L2 Priority 3 is enabled with DC-QCN (ECN). Therefore, RDMA traffic runs on L2 priority 3.
- VLANs are mandatory in this solution.
- Flow Control (or PFC) is not enabled.
- The adapter should be enabled similarly. For example, seeRoCE Configuration for Mellanox Adapters (PCP-Based Resilient).
- For end to end configuration example and troubleshooting, see HowTo Configure Resilient RoCE (ECN) End-to-End Using ConnectX-4 and Spectrum (Trust L2).
1. Enable ECN for RoCE traffic that runs over traffic class 3 in order to get DCQCN congestion control to work.
switch (config) # interface ethernet 1/1-1/32 traffic-class 3 congestion-control ecn minimum-absolute 150 maximum-absolute 1500
Note: For a fair sharing of switch buffer with other traffic classes, it is recommended to configure ECN on all other traffic classes.
2. Configure the buffer pool - allocate buffer pool 0 for lossy traffic.
switch (config) # traffic pool roce type lossyswitch (config) # traffic pool roce memory percent 50.00
switch (config) # traffic pool roce map switch-priority 3
3. Set a strict priority to CNPs over traffic class 6.
interface ethernet 1/1-1/32 traffic-class 6 dcb ets strict
4. Change port mode to hybrid.
Note: This is required for preserving VLAN tag and PCP value when forwarding the packet.
switch (config) # interface ethernet 1/1-1/32 switchport mode hybrid
5. Add the relevant VLAN to the switch.
switch (config) # vlan 10
switch (config vlan 10) exit
switch (config) # interface ethernet 1/1-1/32 switchport hybrid allowed-vlan all
For more information about switch port mode, see HowTo Configure Switch Port Types with MLNX-OS.
For end to end configuration example and troubleshooting, see HowTo Configure Resilient RoCE (ECN) End-to-End Using ConnectX-4 and Spectrum (Trust L2).