RoCE Configuration for Spectrum installed with MLNX-OS (Profile 3)

Version 8

    The following post is shows configuration example RoCE over Mellanox Spectrum switch installed with MLNX-OS. The RoCE is being sent with L3 priority (DSCP) over lossy network and Trust L3.

    The configuration uses DSCP value 26 to classify RoCE traffic and DSCP value 48 to classify CNPs (Congestion Notification Packets) traffic. ECN is enabled on the switch.

    The configuration is described as Profile 3, For other RoCE Profile solutions see Getting Started with RoCE Configuration .

     

    References

     

    Overview

    This solutions has the following network assumption

     

    Configuration

    1. In order to make DCQCN congestion control to work, a user must enable ECN for RoCE traffic that run over traffic class 3:

    switch (config) # interface ethernet 1/1-1/32 traffic-class 3 congestion-control ecn minimum-absolute 150 maximum-absolute 1500

     

    For a fair sharing of switch buffer with other traffic classes It is recommended to configure ECN on all other traffic classes as well.

     

    2. Buffer pool configuration.

    Allocating a buffer pool 0 for background traffic and 1 to RoCE traffic.

    switch (config) # pool ePool0 direction egress-mc size 6904320 type dynamic

    switch (config) # pool iPool0 direction ingress size 6904320 type dynamic

    switch (config) # pool ePool1 direction egress size 6904320 type dynamic

    switch (config) # pool iPool1 direction ingress size 6904320 type dynamic

     

    3. Bind interfaces to switch-priority

    Binding switch priorities 3 and 6 to ingress PG group 3 and 6.

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg6 bind switch-priority 6

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg3 bind switch-priority 3

     

    4. Mapping ingress/egress interface to pool configuration

    Mapping RoCE traffic to shared pool 1

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg3 map pool iPool1 type lossy reserved 20480 shared alpha 8

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg6 map pool iPool1 type lossy reserved 20480 shared alpha 8

     

    5. Setting strict priority to CNPs over traffic class 6

    switch (config) # interface ethernet 1/1-1/32 traffic-class 6 dcb ets strict

     

    6. Set trust layer to L3

    switch (config) # interface ethernet 1/1-1/32 qos trust L3

     

    For end to end configuration example and troubleshooting,  HowTo Configure Resilient RoCE (ECN) End-to-End Using ConnectX-4 and Spectrum (Trust L3)