RoCE Configuration for MLNX-OS Switches in PCP-Based QoS Mode (advanced mode)

Version 12

    Note - this post is relevant for versions older than 3.6.5000 and can also be used to explain advanced mode usage.

    For simple configuration of 3.6.5000 and above see Recommended Network Configuration Examples for RoCE Deployment

     

    This post provides a configuration example of RoCE over lossy networks for MLNX-OS switches in PCP-based QoS mode.

     

    References

     

    Overview

    This solutions offers the following network setup:

     

    Configuration

    1. Enable ECN for RoCE traffic that runs over traffic class 3 in order to get DCQCN congestion control to work.

    switch (config) # interface ethernet 1/1-1/32 traffic-class 3 congestion-control ecn minimum-absolute 150 maximum-absolute 1500

    Note: For a fair sharing of switch buffer with other traffic classes, it is recommended to configure ECN on all other traffic classes.

     

    2. Configure the buffer pool - allocate buffer pool 0 for lossy traffic.

    switch (config) # advanced buffer management force // required for version 3.6.5000 and above

    switch (config) # pool ePool0 size 10485760 type dynamic

    switch (config) # pool iPool0 size 10485760 type dynamic

     

    3. Bind interfaces to switch priority - bind switch priorities 3 and 6 to ingress PG group 3 and 6.

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg6 bind switch-priority 6

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg3 bind switch-priority 3

     

    4. Map ingress/egress interface to pool configuration by allocating buffer to priority 3 and mapping it to a lossless pool, and allocating buffer to priority 6 and mapping it to a lossy pool.

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg3 map pool iPool0 type lossy reserved 20480 shared alpha 8

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg6 map pool iPool0 type lossy reserved 20480 shared alpha 8

     

    5. Set a strict priority to CNPs over traffic class 6.

    interface ethernet 1/1-1/32 traffic-class 6 dcb ets strict

     

    6. Change port mode to hybrid.

    Note: This is required for preserving VLAN tag and PCP value when forwarding the packet.

    switch (config) # interface ethernet 1/1-1/32 switchport mode hybrid

     

    7. Add the relevant VLAN to the switch.

    switch (config) # vlan 10
    switch (config vlan 10) exit
    switch (config) # interface ethernet 1/1-1/32 switchport hybrid allowed-vlan all

     

    For more information about switch port mode, see HowTo Configure Switch Port Types with MLNX-OS.

     

    For end to end configuration example and troubleshooting, see HowTo Configure Resilient RoCE (ECN) End-to-End Using ConnectX-4 and Spectrum (Trust L2).