Lossless RoCE Configuration for MLNX-OS Switches in PCP-Based QoS Mode

Version 3

    This post provides a configuration example of Lossless RoCE for MLNX-OS switch in PCP-based QoS mode.






    This solution offers the following network setup:



    1. Enable ECN for RoCE traffic that runs over traffic class 3 in order to get DCQCN congestion control to work:

    switch (config) # interface ethernet 1/1-1/32 traffic-class 3 congestion-control ecn minimum-absolute 150 maximum-absolute 1500


    Note: For a fair sharing of switch buffer with other traffic classes, it is recommended to configure ECN on all other traffic classes.


    2. Configure the pool for RoCE

    switch (config) # traffic pool roce type lossless

    switch (config) # traffic pool roce memory percent 50.00
    switch (config) # traffic pool roce map switch-priority 3


    3. Set strict priority to CNPs over traffic class 6.

    interface ethernet 1/1-1/32 traffic-class 6 dcb ets strict


    4. Change port mode to hybrid.

    Note: This is required for preserving VLAN tag and PCP value when forwarding the packet.

    switch (config) # interface ethernet 1/1-1/32 switchport mode hybrid


    5. Add the relevant VLAN to the switch.

    switch (config) # vlan 10
    switch (config vlan 10) # exit
    switch (config) # interface ethernet 1/1-1/32 switchport hybrid allow-vlan all

    See more about switch port in HowTo Configure Switch Port Types with MLNX-OS.


    6. Enable PFC on priority 3 on all ports:

    switch (config) # dcb priority-flow-control enable force

    switch (config) # interface ethernet 1/1-1/32 dcb priority-flow-control mode on force


    7. [Optional] Enable DCBX LLDPs.

    Note: This is required for a similar configuration of the adapter card. See RoCE Configuration on Mellanox Adapters (PCP-Based Lossless Traffic).

    switch (config) # lldp


    For end to end configuration example and troubleshooting, see How To Configure Lossless RoCE (PFC + ECN) End-to-End Using ConnectX-4 and Spectrum (Trust L2).