RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode (advanced mode)

Version 16

    Note - this post is relevant for versions older than 3.6.5000 and can also be used to explain advanced mode usage.

    For simple configuration of 3.6.5000 and above see https://community.mellanox.com/docs/DOC-2855

     

    This post provides a configuration example of resilient RoCE for MLNX-OS switches in DSCP-based QoS mode.

    For other configuration modes, see Getting Started with RoCE Configuration.

     

    References

     

    Overview

     

     

    Configuration

    1. Enable ECN for RoCE traffic that runs over traffic class 3:

    switch (config) # interface ethernet 1/1-1/32 traffic-class 3 congestion-control ecn minimum-absolute 150 maximum-absolute 1500

     

    Note: For a fair sharing of switch buffer with other traffic classes, it is recommended to configure ECN on all other traffic classes.

     

    2. Configure the buffer pool - allocate buffer pool 0 for background traffic and 1 to RoCE traffic.

    switch (config) # advanced buffer management force // required for version 3.6.5000 and above

    switch (config) # pool ePool1 size 6904320 type dynamic

    switch (config) # pool iPool1 size 6904320 type dynamic

    switch (config) # pool ePool0 size 6904320 type dynamic

    switch (config) # pool iPool0 size 6904320 type dynamic

    3. Bind interfaces to switch-priority - bind switch priorities 3 and 6 to ingress PG group 3 and 6.

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg6 bind switch-priority 6

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg3 bind switch-priority 3

     

    4. Map ingress/egress interface to pool configuration by mapping RoCE traffic to shared pool 1.

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg3 map pool iPool1 type lossy reserved 20480 shared alpha 8

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg6 map pool iPool1 type lossy reserved 20480 shared alpha 8

     

    5. Set a strict priority to CNPs over traffic class 6.

    switch (config) # interface ethernet 1/1-1/32 traffic-class 6 dcb ets strict

     

    6. Set trust layer to L3.

    switch (config) # interface ethernet 1/1-1/32 qos trust L3

     

    For end to end configuration example and troubleshooting,  HowTo Configure RoCE over a Lossy Fabric (ECN) End-to-End Using ConnectX-4 and Spectrum (Trust L3)