HowTo Configure Flow Control on Cisco 3K C3232C (NX-OS 7.X)

Version 7

    This post discusses how to configure flow control and troubleshoot it on the Cisco Nexus 3K switch.

     

    References

     

    Overview

    Normally, flow control port configuration on the Cisco NX-OS is achieved via a simple set of commands. For example:

    switch (config)# interface Ethernet1/1

    switch (config-if)# flowcontrol receive on

    switch (config-if)# flowcontrol send on

     

    However, when we tested this configuration when there is congestion, we noted that the switch did not appear to send pause frames (Tx pause = 0) on the relevant ports, and the pause was not propagated.

    This information is consistent with the standard manual.

     

    Configuration

    To resolve this issue and be able to use flow control, you need to specify Network-QoS policies and classes, and then indicate that they are to be paused on CoS 0 as follows:

    1. Enable flow control on the relevant interfaces:

    switch (config)# interface Ethernet1/1

    switch (config-if)# flowcontrol receive on

    switch (config-if)# flowcontrol send on

     

    2. Configure for Policy Mapping and map it to the System QoS as follows.

    switch (config)# class-map type qos RDMA                            <-- RDMA is just the name of the class

    switch (config-cmap-qos)# match cos 0                               <-- Classify all traffic with CoS 0, in this case untagged traffic.

    switch (config-cmap-qos)# exit

     

    switch (config)# policy-map type network-qos QOS_NETWORK            <-- QOS_NETWORK is just the name of the policy

    switch (config-pmap-nqos)#   class type network-qos RDMA            <-- Use the RDMA class as the traffic that will be policed in this case.

    switch (config-pmap-nqos-c)# pause pfc-cos 0                        <-- Pause all classified traffic that has CoS 0

    switch (config-pmap-nqos-c)# exit

    switch (config-pmap-nqos)# exit

     

    switch (config)#  system qos

    switch (config )#    service-policy type network-qos QOS_NETWORK    <-- Apply QOS_NETWORK policy to the switch (all ports)

     

    Verification

    1. Generate congestion on a specific port, make sure that the traffic is populated, and verify that the other Cisco port is sending pause frames.

    In this case ports 1/20 and 1/22/1 were used and the pause was populated:

    switch (config)# show interface flowcontrol

     

    --------------------------------------------------------------------------------

    Port         Send FlowControl  Receive FlowControl  RxPause   TxPause 

                 admin    oper     admin    oper

    --------------------------------------------------------------------------------

    Eth1/20      on       on       on       on          2         0       

    Eth1/21      off      off      off      off         0         0       

    Eth1/22/1    on       on       on       on          0         10676578

     

     

    2. Check the Queuing counters and search for dropped packets. In this case, we are viewing data for QoS Group 0:

     

    # show queuing interface ethernet 1/22/1

     

    slot  1

    =======

     

    Egress Queuing for Ethernet1/22/1 [System]

    ------------------------------------------------------------------------------

    QoS-Group# Bandwidth% PrioLevel                Shape                   QLimit

                                       Min          Max        Units  

    ------------------------------------------------------------------------------

          3             -         1           -            -     -            6(D)

          2             0         -           -            -     -            6(D)

          1             0         -           -            -     -            6(D)

          0           100         -           -            -     -            -(U)

    +-------------------------------------------------------------------+

    |                              QOS GROUP 0                          |

    +-------------------------------------------------------------------+

    |                |  Unicast       | OOBFC Unicast  |  Multicast     |

    +-------------------------------------------------------------------+

    |        Tx Pkts |        57467748|               0|               4|

    |        Tx Byts |      4482486329|               0|             256|

    |   Dropped Pkts |               0|               0|               0|

    |   Dropped Byts |               0|               0|               0|

    |   Q Depth Byts |               0|               0|               0|

    +-------------------------------------------------------------------+

    |                              QOS GROUP 1                          |

    +-------------------------------------------------------------------+

    |                |  Unicast       | OOBFC Unicast  |  Multicast     |

    +-------------------------------------------------------------------+

    |        Tx Pkts |               0|               0|               0|

    |        Tx Byts |               0|               0|               0|

    |   Dropped Pkts |               0|               0|               0|

    |   Dropped Byts |               0|               0|               0|

    |   Q Depth Byts |               0|               0|               0|

    +-------------------------------------------------------------------+

    .

    .

    .

     

    +-------------------------------------------------------------------+

    |                              QOS GROUP 7                          |

    +-------------------------------------------------------------------+

    |                |  Unicast       | OOBFC Unicast  |  Multicast     |

    +-------------------------------------------------------------------+

    |        Tx Pkts |               0|               0|               0|

    |        Tx Byts |               0|               0|               0|

    |   Dropped Pkts |               0|               0|               0|

    |   Dropped Byts |               0|               0|               0|

    |   Q Depth Byts |               0|               0|               0|

    +-------------------------------------------------------------------+

    |                      CONTROL QOS GROUP                            |

    +-------------------------------------------------------------------+

    |                |  Unicast       | OOBFC Unicast  |  Multicast     |

    +-------------------------------------------------------------------+

    |        Tx Pkts |          178571|               0|               0|

    |        Tx Byts |        12627032|               0|               0|

    |   Dropped Pkts |               0|               0|               0|

    |   Dropped Byts |               0|               0|               0|

    |   Q Depth Byts |               0|               0|               0|

    +-------------------------------------------------------------------+

    |                         SPAN QOS GROUP                            |

    +-------------------------------------------------------------------+

    |                |  Unicast       | OOBFC Unicast  |  Multicast     |

    +-------------------------------------------------------------------+

    |        Tx Pkts |               0|               0|               0|

    |        Tx Byts |               0|               0|               0|

    |   Dropped Pkts |               0|               0|               0|

    |   Dropped Byts |               0|               0|               0|

    |   Q Depth Byts |               0|               0|               0|

    +-------------------------------------------------------------------+

     

     

    Port Egress Statistics

    --------------------------------------------------------

    WRED Drop Pkts                              0

     

    Ingress Queuing for Ethernet1/22/1

    ------------------------------------------------------------------

    QoS-Group#                 Pause                        QLimit

               Buff Size       Pause Th      Resume Th     

    ------------------------------------------------------------------

          7              -            -            -           10(D)

          6              -            -            -           10(D)

          5              -            -            -           10(D)

          4              -            -            -           10(D)

          3              -            -            -           10(D)

          2              -            -            -           10(D)

          1              -            -            -           10(D)

         0          46450        25475        20975            4(D)

     

     

     

     

    Port Ingress Statistics

    --------------------------------------------------------

    Ingress MMU Drop Pkts                               0

    Ingress MMU Drop Bytes                              0

     

     

    PFC Statistics

    ----------------------------------------------------------------------------

    TxPPP:                    0, RxPPP:                    0

    ----------------------------------------------------------------------------

    COS QOS Group        PG   TxPause   TxCount         RxPause         RxCount

       0         0         7  Inactive         0        Inactive               0

       1         -         7  Inactive         0        Inactive               0

       2         -         7  Inactive         0        Inactive               0

       3         -         7  Inactive         0        Inactive               0

       4         -         7  Inactive         0        Inactive               0

       5         -         7  Inactive         0        Inactive               0

       6         -         7  Inactive         0        Inactive               0

       7         -         7  Inactive         0        Inactive               0

    ----------------------------------------------------------------------------