Lossless RoCE Configuration for MLNX-OS Switches in PCP-Based QoS Mode (advanced mode)

Version 19

    Note - WRT shared buffers configuration this post is relevant for versions older than 3.6.5000 and can also be used to explain advanced mode usage.

               For simple configuration of 3.6.5000 and above see https://community.mellanox.com/docs/DOC-2855

     

    This post provides a configuration example of Lossless RoCE for MLNX-OS switch in PCP-based QoS mode.

    For other configuration modes, see Getting Started with RoCE Configuration.

     

    References

     

    Overview

    This solution offers the following network setup:

     

    Configuration

    1. Enable ECN for RoCE traffic that runs over traffic class 3 in order to get DCQCN congestion control to work:

    switch (config) # interface ethernet 1/1-1/32 traffic-class 3 congestion-control ecn minimum-absolute 150 maximum-absolute 1500

     

    Note: For a fair sharing of switch buffer with other traffic classes, it is recommended to configure ECN on all other traffic classes.

     

    2. Configure the buffer pool - allocate buffer pool 0 for lossy traffic and pool 1 for lossless traffic.

    switch (config) # advanced buffer management force // required for version 3.6.5000 and above

    switch (config) # pool ePool0 size 5242880 type dynamic

    switch (config) # pool iPool0 size 5242880 type dynamic

    switch (config) # pool ePool1 size 16777000 type dynamic

    switch (config) # pool iPool1 size 5242880 type dynamic

     

    3. Bind interfaces to switch-priority - bind switch priorities 3 and 6 to ingress PG group 3 and 6.

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg6 bind switch-priority 6

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg3 bind switch-priority 3

     

    4. Map ingress/egress interface to pool configuration by allocating buffer to priority 3 and mapping it to a lossless pool, and allocating buffer to priority 6 and mapping it to a lossy pool.

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg3 map pool iPool1 type lossless reserved 67538 xoff 18432 xon 18432 shared alpha 2

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg6 map pool iPool0 type lossy reserved 10240 shared alpha 8

    switch (config) # interface ethernet 1/1-1/32 egress-buffer ePort.tc3 map pool ePool1 reserved 1500 shared alpha inf

     

    5. Set strict priority to CNPs over traffic class 6.

    interface ethernet 1/1-1/32 traffic-class 6 dcb ets strict

     

    6. Change port mode to hybrid.

    Note: This is required for preserving VLAN tag and PCP value when forwarding the packet.

    switch (config) # interface ethernet 1/1-1/32 switchport mode hybrid

     

    7. Add the relevant VLAN to the switch.

    switch (config) # vlan 10
    switch (config vlan 10) # exit
    switch (config) # interface ethernet 1/1-1/32 switchport hybrid allow-vlan all

    See more about switch port in HowTo Configure Switch Port Types with MLNX-OS.

     

    8. Enable PFC on priority 3 on all ports:

    switch (config) # dcb priority-flow-control enable force

    switch (config) # dcb priority-flow-control priority 3 enable

    switch (config) # interface ethernet 1/1-1/32 dcb priority-flow-control mode on force

     

    9. Enable DCBX LLDPs.

    Note: This is required for a similar configuration of the adapter card. See RoCE Configuration on Mellanox Adapters (PCP-Based Lossless Traffic).

    switch (config) # lldp

     

    For end to end configuration example and troubleshooting, see How To Configure Lossless RoCE (PFC + ECN) End-to-End Using ConnectX-4 and Spectrum (Trust L2).