HowTo Configure PFC on Cisco 3K C3232C (NX-OS 7.X)

Version 9

    This post shows the basic procedure used to enable Priority Flow Control (PFC) on a Cisco switch C3232C.

    This procedure was tested with Cisco C3232C, version NXOS: version 7.0(3)IX1(2).

     

    References

     

    Setup

    Connect two or more hosts to the switch. Install Mellanox ConnectX-3 or ConnectX-4 adapters on the hosts. In this example, we will use VLAN 100, and L2 priority 4.

     

     

    Configuration

    1. Configure PFC on the hosts. Refer to the HowTo Run RoCE over L2 Enabled with PFC, or HowTo Configure PFC on ConnectX-4 posts for configuration instructions.

     

    2. Create VLAN.

    switch (config) # vlan 100

     

    3. Specify the interface switchport mode as trunk (run this on all relevant interfaces).

    switch (config) # interface ethernet 1/20

    switch (config-if) # switchport mode trunk

     

    4. Enable PFC on the relevant interfaces as follows:

    switch (config) # interface ethernet 1/20

    switch (config-if) # priority-flow-control mode on

     

    5. (Optional) Set MTU to Jumbo (or any other size, as needed for your application).

    switch (config) # interface ethernet 1/20

    switch (config-if) # mtu 9000

     

    6. Configure the TCAM region (needed for the qos commands)

    switch (config) # hardware access-list tcam region racl-lite 512

    switch (config) # hardware access-list tcam region qos-intra-lite 512

     

    7. Create a network-qos class-map (to specify how classification rules should be used).

    switch (config) # class-map type network-qos pfc_class

    switch (config-cmap-nqos) # match qos-group 4

     

    8. Create a network-qos policy based on the class-map created in the previous step (pfc_class).

    Pause priority (CoS) 4, and optionally set the MTU for that class.

    switch (config) # policy-map type network-qos pfc_policy

    switch (config-pmap-nqos) # class type network-qos pfc_class

    switch (config-pmap-nqos-c) # pause pfc-cos 4

    switch (config-pmap-nqos-c) # mtu 9000

     

    9. Set the System with the network-qos policy

    switch (config) # system qos

    switch (config-sys-qos) #service-policy type network-qos pfc_policy

     

    10. Create a qos class

    Note: there is a difference between qos and network-qos classification options.

    switch (config) # class-map type qos qos_class

    switch (config-cmap-qos) # match cos 4

     

    11. Create a qos policy based on the class-map created in the previous step (qos_class).

    switch (config) #  policy-map type qos qos_policy

    switch (config-pmap-qos) #  class qos_class

    switch (config-pmap-qos-c) #  set qos-group 4

    switch (config-pmap-qos-c) #  set cos 4

     

    12. Per interface, map the qos policy just created.

    switch (config) # interface ethernet 1/20

    switch (config-if) #  service-policy type qos input qos_policy

    Make sure to add this to all interfaces.

     

    13. Reserve space (in percentage) from the shared buffer to the PFC traffic (for details click here)

    switch (config) # hardware profile pfc mmu buffer-reservation 48

     

    Verification

     

    1. Run the benchmark test, for example ib_write_bw, between the two servers, and generate some congestion. In this case we used the ConnectX-4 switch configured to operate on 50G and sent RDMA over Converged Ethernet (RoCE) traffic to the ConnectX-3 adapter 40G.

    See the verification examples, in HowTo Configure PFC on ConnectX-4 .

     

    In this case, port 1/20 was 40G, while port 1/22/1 was 50G.

     

    2. Display the QoS running-config what configuration is running to verify.

    # show running-config ipqos

     

    version 7.0(3)I3(1)

     

    class-map type qos match-all qos_class

      match cos 4

     

    policy-map type qos qos_policy

      class qos_class

        set qos-group 4

        set cos 4

     

     

    class-map type network-qos pfc_class

    match qos-group 4

     

    policy-map type network-qos pfc_policy

      class type network-qos pfc_class

        pause pfc-cos 4

        mtu 9000

     

    system qos

      service-policy type network-qos pfc_policy

     

    interface Ethernet1/20

      priority-flow-control mode on

        service-policy type qos input qos_policy

     

    interface Ethernet1/22/1

      priority-flow-control mode on

      service-policy type qos input qos_policy

     

    3. Clear all counters as follows:

    switch (config)# clear counters

    switch (config)# clear qos statistics

     

    4.  Check that there are PFC pause frames on the 40G interface (you should expect to see Rx pause frames on priority 4) and verify that the administrative (admin) and operating (oper) states of the PFC modes are set on.

    Run:

    switch (config) # show interface ethernet 1/20 priority-flow-control

     

    slot  1

    =======

     

    ============================================================

    Port               Mode Oper(VL bmap)  RxPPP      TxPPP    

    ============================================================

     

    Ethernet1/20       On   On  (10)      3964      0         

     

     

    switch (config)# show interface ethernet 1/20 priority-flow-control detail

     

    slot  1

    =======

    Ethernet1/20

        Admin Mode: On 

        Oper Mode: On 

        VL bitmap: (10)     

        Total Rx PFC Frames: 4078     

        Total Tx PFC Frames: 0        

        ---------------------------------------------------------------------------------------------------------------------

            |  Priority0  |  Priority1  |  Priority2  |  Priority3  |  Priority4  |  Priority5  |  Priority6  |  Priority7  |

        ---------------------------------------------------------------------------------------------------------------------

        Rx  |0            |0            |0            |0            |4078         |0            |0            |0           

        ---------------------------------------------------------------------------------------------------------------------

        Tx  |0            |0            |0            |0            |0            |0            |0            |0           

     

     

    5. Check to see if there are any PFC priority pause frames on the 50G interface (you should expect to see Tx pause frames on priority 4) and verify that the admin and oper states of the PFC are set on.

    # show interface ethernet 1/22/1 priority-flow-control

     

    slot  1

    =======

     

    ============================================================

    Port               Mode Oper(VL bmap)  RxPPP      TxPPP    

    ============================================================

    Ethernet1/22/1     On   On  (10)      0          5114778   

     

    switch (config)# show interface ethernet 1/22/1 priority-flow-control detail

     

    slot  1

    =======

    Ethernet1/22/1

        Admin Mode: On 

        Oper Mode: On 

        VL bitmap: (10)     

        Total Rx PFC Frames: 0        

        Total Tx PFC Frames: 5114778  

        ---------------------------------------------------------------------------------------------------------------------

            |  Priority0  |  Priority1  |  Priority2  |  Priority3  |  Priority4  |  Priority5  |  Priority6  |  Priority7  |

        ---------------------------------------------------------------------------------------------------------------------

        Rx  |0            |0            |0            |0            |0            |0            |0            |0           

        ---------------------------------------------------------------------------------------------------------------------

        Tx  |0            |0            |0            |0            |5114778      |0            |0            |0           

     

    6. Verify that there are no input or output discard frames on each of the ports.

    Here is an example of what you would see on the 40G port:

    switch (config) # show interface ethernet 1/20

    Ethernet1/20 is up

    admin state is up, Dedicated Interface

      Hardware: 40000/100000 Ethernet, address: cc46.d664.1c10 (bia cc46.d664.1c10)

      MTU 9000 bytes, BW 40000000 Kbit, DLY 10 usec

      reliability 255/255, txload 1/255, rxload 1/255

      Encapsulation ARPA, medium is broadcast

      Port mode is trunk

      full-duplex, 40 Gb/s, media type is 40G

      Beacon is turned off

      Auto-Negotiation is turned on

      Input flow-control is off, output flow-control is off

      Auto-mdix is turned on

      Rate mode is dedicated

      Switchport monitor is off

      EtherType is 0x8100

      EEE (efficient-ethernet) : n/a

      Last link flapped 20:44:31

      Last clearing of "show interface" counters 00:07:41

      0 interface resets

      30 seconds input rate 0 bits/sec, 0 packets/sec

      30 seconds output rate 0 bits/sec, 7 packets/sec

      Load-Interval #2: 5 minute (300 seconds)

        input rate 12.07 Mbps, 18.32 Kpps; output rate 6.95 Gbps, 788.09 Kpps

      RX

        10392558 unicast packets  0 multicast packets  1 broadcast packets

        10392559 input packets  852756075 bytes

        0 jumbo packets  0 storm suppression packets

        0 runts  0 giants  0 CRC  0 no buffer

        0 input error  0 short frame  0 overrun   0 underrun  0 ignored

        0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

        0 input with dribble  0 input discard

        0 Rx pause

      TX

        445533763 unicast packets  3975 multicast packets  259 broadcast packets

        445537997 output packets  491089886408 bytes

        0 jumbo packets

        0 output error  0 collision  0 deferred  0 late collision

        0 lost carrier  0 no carrier  0 babble  0 output discard

        0 Tx pause

     

    7.  Check the queuing status and counters.

    - Make sure that no drops or pause frames are being sent.

    - Make sure to see traffic on QoS Group 4 (RoCE traffic).

     

    switch (config) # show queuing interface ethernet 1/20 summary

     

    slot  1

    =======

     

    Egress Queuing for Ethernet1/20 [System]

    ------------------------------------------------------------------------------

    QoS-Group# Bandwidth% PrioLevel                Shape                   QLimit

                                       Min          Max        Units  

    ------------------------------------------------------------------------------

          3             -         1           -            -     -            6(D)

          2             0         -           -            -     -            6(D)

          1             0         -           -            -     -            6(D)

          0           100         -           -            -     -            6(D)

    +-------------------------------------------------------------------+

    |                              QOS GROUP 0                          |

    +-------------------------------------------------------------------+

    |                |  Unicast       | OOBFC Unicast  |  Multicast     |

    +-------------------------------------------------------------------+

    |        Tx Pkts |               0|               0|               0|

    |        Tx Byts |               0|               0|               0|

    |   Dropped Pkts |               0|               0|               0|

    |   Dropped Byts |               0|               0|               0|

    |   Q Depth Byts |               0|               0|               0|

    +-------------------------------------------------------------------+

    ...

    ...

    +-------------------------------------------------------------------+

    |                              QOS GROUP 4                          |

    +-------------------------------------------------------------------+

    |                |  Unicast       | OOBFC Unicast  |  Multicast     |

    +-------------------------------------------------------------------+

    |        Tx Pkts |        64575649|               0|               0|

    |        Tx Byts |      5295203218|               0|               0|

    |   Dropped Pkts |               0|               0|               0|

    |   Dropped Byts |               0|               0|               0|

    |   Q Depth Byts |               0|               0|               0|

    +-------------------------------------------------------------------+

    ...

    ...

    +-------------------------------------------------------------------+

    |                      CONTROL QOS GROUP                            |

    +-------------------------------------------------------------------+

    |                |  Unicast       | OOBFC Unicast  |  Multicast     |

    +-------------------------------------------------------------------+

    |        Tx Pkts |            5247|               0|               5|

    |        Tx Byts |          376266|               0|             340|

    |   Dropped Pkts |               0|               0|               0|

    |   Dropped Byts |               0|               0|               0|

    |   Q Depth Byts |               0|               0|               0|

    +-------------------------------------------------------------------+

    |                         SPAN QOS GROUP                            |

    +-------------------------------------------------------------------+

    |                |  Unicast       | OOBFC Unicast  |  Multicast     |

    +-------------------------------------------------------------------+

    |        Tx Pkts |               0|               0|               0|

    |        Tx Byts |               0|               0|               0|

    |   Dropped Pkts |               0|               0|               0|

    |   Dropped Byts |               0|               0|               0|

    |   Q Depth Byts |               0|               0|               0|

    +-------------------------------------------------------------------+

     

     

    Port Egress Statistics

    --------------------------------------------------------

    WRED Drop Pkts                              0

     

    Ingress Queuing for Ethernet1/20

    ------------------------------------------------------------------

    QoS-Group#                 Pause                        QLimit

               Buff Size       Pause Th      Resume Th     

    ------------------------------------------------------------------

          7              -            -            -           10(D)

          6              -            -            -           10(D)

          5              -            -            -           10(D)

         4         101610        64305        37305            4(D)

          3              -            -            -           10(D)

          2              -            -            -           10(D)

          1              -            -            -           10(D)

          0              -            -            -           10(D)

     

    Port Ingress Statistics

    --------------------------------------------------------

    Ingress MMU Drop Pkts                               0

    Ingress MMU Drop Bytes                              0

     

    PFC Statistics

    ----------------------------------------------------------------------------

    TxPPP:                    0, RxPPP:                 8842

    ----------------------------------------------------------------------------

    COS QOS Group        PG   TxPause   TxCount         RxPause         RxCount

       0         -         -  Inactive         0        Inactive               0

       1         -         -  Inactive         0        Inactive               0

       2         -         -  Inactive         0        Inactive               0

       3         -         -  Inactive         0        Inactive               0

       4         4         1  Inactive         0        Inactive            8842

       5         -         -  Inactive         0        Inactive               0

       6         -         -  Inactive         0        Inactive               0

       7         -         -  Inactive         0        Inactive               0

    ----------------------------------------------------------------------------

     

    Troubleshooting and Open Issues

     

    1. There could be other commands that could be added to the switch as workarounds, but they were not investigated. The following are such examples:

    switch (config) # hardware access-list tcam region racl-lite 512

    switch (config) # hardware access-list tcam region qos-intra-lite 512

     

    More investigation should be done on this case.

     

    2. Make sure to use the latest switch software, in older version, the qos mapping was not applicable.