This post provides a configuration example of lossless RoCE for MLNX-OS switches in DSCP-based QoS mode.
This post assumes MLNX-OS version 3.6.5000 and above. For manual buffer configuration and older versions see: Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode (advanced mode)
For other configuration modes, see Getting Started with RoCE Configuration.
- Getting Started with RoCE Configuration
- Ethernet Switch Solutions
- HowTo Configure ECN on Mellanox Ethernet Switches (Spectrum)
- Lossless RoCE Configuration for Linux Drivers in DSCP-Based QoS Mode
This solution offers the following network setup:
- RoCE traffic uses DSCP 26.
- CNP traffic uses DSCP 48.
- PFC is enabled on the adapter card over priority 3 (DSCP 26 is mapped to priority 3). See Lossless RoCE Configuration for Linux Drivers in DSCP-Based QoS Mode.
1. Enable ECN for RoCE traffic over traffic class 3.
Traffic over DSCP 26 is mapped to traffic class 3 by default.
switch (config) # interface ethernet 1/1-1/32 traffic-class 3 congestion-control ecn minimum-absolute 150 maximum-absolute 1500
Note: If TCP traffic runs over other traffic class, it is recommended to configure ECN on it.
2. Configure the pool for RoCE
Note: In this example, the shared space is split equally between the RoCE pool and non-RoCE pool. If the network has a different ratio of RoCE/non-RoCE traffic, the shared space can be divided accordingly.
switch (config) # traffic pool roce type losslessswitch (config) # traffic pool roce memory percent 50.00
switch (config) # traffic pool roce map switch-priority 3
3. Set a strict priority to CNPs over traffic class 6.
Traffic over DSCP 48 is mapped to switch-priority 6 by default.
Note: In this example equal weighted round robin scheduling is used between RoCE and non-RoCE traffic (which is set according to switch defaults, hence no additional commands are required). If the network has a different ratio of RoCE/non-RoCE traffic, the round-robin weights can be set accordingly.
switch (config) # interface ethernet 1/1-1/32 traffic-class 6 dcb ets strict
4. Set trust mode L3 (DSCP).
switch (config) # interface ethernet 1/1-1/32 qos trust L3
5. Enable receive PFC on priority 3 on all ports.
switch (config) # dcb priority-flow-control enable force
switch (config) # interface ethernet 1/1-1/32 dcb priority-flow-control mode on force
6. [Optional] Enable DCBX LLDP.
Note: This is required in case the adapter card relies on LLDP configuration in the switch for setting priority for PFC. See Lossless RoCE Configuration for Linux Drivers in DSCP-Based QoS Mode.
switch (config) #lldp