Lossless RoCE Configuration for MLNX-OS switch in PCP-based QoS mode

Version 15

    The following post shows configuration example of Lossless RoCE for MLNX-OS switch in PCP-based QoS mode.

    For other configuration modes, see Getting Started with RoCE Configuration .

     

    References

     

    Overview

    This solutions has the following network assumption

     

    Configuration

    1. In order to make DCQCN congestion control to work, a user must enable ECN for RoCE traffic that run over traffic class 3:

    switch (config) # interface ethernet 1/1-1/32 traffic-class 3 congestion-control ecn minimum-absolute 150 maximum-absolute 1500

     

    For a fair sharing of switch buffer with other traffic classes It is recommended to configure ECN on all other traffic classes as well.

     

    2. Buffer pool configuration.

    Allocating a buffer pool 0 for lossy traffic and pool 1 for lossless traffic.

    switch (config) # pool ePool0 direction egress size 5242880 type dynamic

    switch (config) # pool iPool0 direction ingress size 5242880 type dynamic

    switch (config) # pool ePool1 direction egress-mc size 16777000 type dynamic

    switch (config) # pool iPool1 direction ingress size 5242880 type dynamic

     

    3. Bind interfaces to switch-priority

    Binding switch priorities 3 and 6 to ingress PG group 3 and 6.

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg6 bind switch-priority 6

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg3 bind switch-priority 3

     

    4. Mapping ingress/egress interface to pool configuration

    Allocating buffer to priority 3 and mapping it to a lossless pool and allocating buffer to priority 6 and mapping it to a lossy pool:

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg3 map pool iPool1 type lossless reserved 67538 xoff 18432 xon 18432 shared alpha 2

    switch (config) # interface ethernet 1/1-1/32 ingress-buffer iPort.pg6 map pool iPool0 type lossy reserved 10240 shared alpha 8

    switch (config) # interface ethernet 1/1-1/32 egress-buffer ePort.tc3 map pool ePool1 reserved 1500 shared alpha inf

     

    5. Setting strict priority to CNPs over traffic class 6

    interface ethernet 1/1-1/32 traffic-class 6 dcb ets strict

     

    6. Change port mode to hybrid.

    Required for preserving VLAN tag and PCP value when forwarding the packet.

    switch (config) # interface ethernet 1/1-1/32 switchport mode hybrid

     

    7. Add relevant VLAN to the switch.

    switch (config) # vlan 10
    switch (config vlan 10) # exit
    switch (config) # interface ethernet 1/1-1/32 switchport hybrid allow-vlan all

    See more about switchport in HowTo Configure Switch Port Types with MLNX-OS.

     

    8. Enable PFC on priority 3 on all ports:

    switch (config) # dcb priority-flow-control enable force

    switch (config) # dcb priority-flow-control priority 3 enable

    switch (config) # interface ethernet 1/1-1/32 dcb priority-flow-control mode on force

     

    9. Enable DCBX LLDPs

    Required for similar configuration of the NIC, see RoCE Configuration for Mellanox Adapters (PCP-based lossless)

    switch (config) # lldp

     

    For end to end configuration example and troubleshooting, see How To Configure Lossless RoCE (PFC + ECN) End-to-End Using ConnectX-4 and Spectrum (Trust L2)