HowTo Enable PFC on Mellanox Switches (SwitchX)

Version 12

    This post demonstrates how to enable PFC on Mellanox SwitchX based switches.

    Note: In case you have Spectrum based switch, refer to How to Enable PFC on Mellanox Switches (Spectrum) .

     

    References

     

    Configuration

    1. Create a VLAN and set a switchport in hybrid (or trunk) mode. Run:

    switch (config) # vlan 100

    switch (config vlan 100) # exit

    switch (config) # interface ethernet 1/1 switchport mode hybrid

    switch (config) # interface ethernet 1/1 switchport hybrid allowed-vlan all

    switch (config) # interface ethernet 1/2 switchport mode hybrid

    switch (config) # interface ethernet 1/2 switchport hybrid allowed-vlan all

     

    2. Make sure Flow Control is disabled (it is disabled by default). Run:

    switch (config) # interface ethernet 1/1-1/2 flowcontrol send off force

    switch (config) # interface ethernet 1/1-1/2 flowcontrol receive off force

     

    3. Enable PFC on the desired priority. Run:

    switch (config) # dcb priority-flow-control enable

    This action might cause traffic loss while shutting down a port with priority-flow-control mode on

    Type 'yes' to confirm  enable pfc globally: yes

    switch (config) # dcb priority-flow-control priority 3 enable
    switch (config) # interface ethernet 1/1-1/2 dcb priority-flow-control mode on force

     

    Verification

    1. Verify PFC configuration. Run:

    switch (config)# show dcb priority-flow-control

    PFC enabled

    Pri

    ority Enabled List   :3

    Priority Disabled List 
    0 1 2 4 5 6 7

    TC     Lossless

    ---    ----------

    0           N

    1           Y

    2           Y

    3           N

    Interface      PFC admin        PFC oper

    ------------  --------------   -------------

    1/1            On               Enabled

    1/2            On               Enabled

    switch (config) #

     

    2. For troubleshooting, check port priority counters.

    For example in case you use Mellanox Ethernet Switches (MLNX-OS) run the following command to check the PFC counters.

    Make sure that pause packets are being sent or received on the proper priority. In normal cases you can expect pause packets to be sent from the receiver side via the switch back to the sender side.

    In this example, Server 1 is the sender, and is connected via switch port number 1/1 while Server 2 is the receiver and is connected via switch port number 1/2.

    You can see that Server 2 is sending pause frames to the switch, and that the switch passes them on via port 1 to Server 1.

     

    Run:

    switch (config) # show interfaces ethernet 1/1 counters priority 3

     

    Rx

      11809280             packets

      11809280             unicast packets

      0                    multicast packets

      0                    broadcast packets

      12238754633          bytes

      0                    pause packets

      0                    pause duration milliseconds

     

    Tx

      2046842              packets

      2046841              unicast packets

      0                    multicast packets

      1                    broadcast packets

      287974440            bytes

      53558                pause packets

     

    switch (config) # show interfaces ethernet 1/2 counters priority 3

     

    Rx

      2046842              packets

      2046841              unicast packets

      0                    multicast packets

      1                    broadcast packets

      247037600            bytes

      56966                pause packets

      1621                 pause duration milliseconds

     

    Tx

      11809283             packets

      11809280             unicast packets

      3                    multicast packets

      0                    broadcast packets

      12474940827          bytes

      0                    pause packets

    switch (config) #