This post provides a configuration example for Mellanox Spectrum installed with MLNX-OS and running RoCE over a lossless network, in PCP-based QoS mode.
- Mellanox adapters and switches support DSCP based QoS and flow control, which is easier and simpler to configure and doesn't require VLANS, QoS is maintained across routers.
- This post assumes MLNX-OS version 3.6.5000 and above. For manual buffer configuration and older versions see: Lossless RoCE Configuration for MLNX-OS Switches in PCP-Based QoS Mode (advanced mode)
Step 1 - Enable ECN Marking
Note - For a fair sharing of switch buffer with other traffic classes, it is recommended to configure ECN on all other traffic classes.
switch (config) # interface ethernet 1/1-1/32 traffic-class 3 congestion-control ecn minimum-absolute 150 maximum-absolute 1500
Step 2 - Create the RoCE pool and set QoS
Configure the traffic pool for RoCE
switch (config) # traffic pool roce type losslessswitch (config) # traffic pool roce memory percent 50.00
switch (config) # traffic pool roce map switch-priority 3
Set a strict priority to CNPs over traffic class 6
switch (config) # interface ethernet 1/1-1/32 traffic-class 6 dcb ets strict
[Optional] Enable DCBX LLDP
Note: This is required in case the adapter card relies on LLDP configuration in the switch for setting priority for PFC
switch (config) #lldp
Step 3 - Per port configuration
Configure the switchport
interface ethernet 1/1-1/32 switchport mode trunk
interface ethernet 1/1-1/32 switchport trunk allowed-vlan [vlan-id]