QoS Configuration examples for Cisco Nexus 5600

Version 11

    This post shows various of QoS configuration required for RoCE applications.

    This post is basic and meant for IT managers.

     

    References

     

    VLAN Configuration

    switch (config) # vlan 100

     

    PFC Configuration (per interface)

    switch (config) # interface ethernet 1/1

    switch (config-if) # switchport mode trunk

    switch (config-if) # priority-flow-control mode on

     

    QoS Configuration

    This procedure performs the following:

    • Classify traffic with CoS 4 and DSCP 34 to RoCE QoS Class (specific class),
    • Map this traffic to queue designated queue.
    • Apply 50% of the bandwidth to RoCE traffic, and leave 50% for the default traffic.
    • Set the class to be non-drop for CoS 4.
    • Set the CoS to be 4 on the egress traffic.
    • Set the MTU to 9216.
    • Apply the policies globally per system.

     

    1. Create qos class-map to map RoCE traffic:

    switch (config) # class-map type qos match-all RoCE_qos_class

    switch (config-cmap-qos) # match cos 4

    switch (config-cmap-qos) # match dscp 34 --> This command is important only for RoCEv2 (to map the traffic between subnets)

    switch (config-cmap-qos) # exit

     

    2. Create a policy map for RoCE traffic:

    switch (config) # policy-map type qos RoCE_qos_policy

    switch (config-pmap-qos) # class RoCE_qos_class

    switch (config-pmap-c-qos) # set qos-group 4

    switch (config-pmap-c-qos) # exit

    switch (config-pmap-qos) # exit

     

    3. Create queuing class map for RoCE traffic:

    switch (config) # class-map type queuing RoCE_queuing_class

    switch (config-cmap-que) # match qos-group 4

    switch (config-cmap-que) # exit

     

    4. Set bandwidth allocation for RoCE traffic:

    switch (config) # policy-map type queuing RoCE_queuing_policy

    switch (config-pmap-que) # class type queuing RoCE_queuing_class

    switch (config-pmap-c-que) # bandwidth percent 50

    switch (config-pmap-c-que) # exit

    switch (config-pmap-que) # class type queuing class-default

    switch (config-pmap-c-que) # bandwidth percent 50

    switch (config-pmap-c-que) # exit

    switch (config-pmap-que) # exit

     

    5. Set Class network QoS for RoCE traffic:

    switch (config) # class-map type network-qos RoCE_network_class

    switch (config-cmap-nq) # match qos-group 4

    switch (config-cmap-nq) # exit

     

    6. Create a policy for network-qos

    switch (config) # policy-map type network-qos RoCE_network_policy

    switch (config-cmap-nq) # class type network-qos RoCE_network_class

    switch (config-cmap-nq-c) # pause no-drop

    switch (config-cmap-nq-c) # mtu 9216

    switch (config-cmap-nq-c) # set cos 4   --> This command is important only for RoCEv2 (to map the traffic between subnets)

    switch (config-cmap-nq-c) # exit

    switch (config-cmap-nq) # class type network-qos class-default

    switch (config-cmap-nq-c) # mtu 9216

    switch (config-cmap-nq-c) # exit

    switch (config-cmap-nq) # exit

     

    7. Create System QoS policy to apply it on the switch (globally on all ports)

    switch (config) # system qos

    switch (config-sys-qos) # service-policy type qos input RoCE_qos_policy

    switch (config-sys-qos) # service-policy type queuing input RoCE_queuing_policy

    switch (config-sys-qos) # service-policy type queuing output RoCE_queuing_policy

    switch (config-sys-qos) # service-policy type network-qos RoCE_network_policy

    switch (config-sys-qos) # exit

     

    8. ECN configuration

    switch (config) # hardware random-detect min-thresh 10g 64000 40g 4000 max-thresh 10g 128000 40g 256000 ecn qos-group 4

    switch (config) # hardware pq-drain 10g 9900 40g 39900

     

    Show commands

    switch (config) # show queuing interface ethernet 1/1

    Ethernet1/1 queuing information:

      TX Queuing

        qos-group  sched-type  oper-bandwidth

            0       WRR             50

            4       WRR             50

     

     

      RX Queuing

        qos-group 0

        q-size: 100160, q-size-40g: 100160, HW MTU: 9216 (9216 configured)

        drop-type: drop, xon: 0, xoff: 0

        Statistics:

            Pkts received over the port             : 7222

            Ucast pkts sent to the cross-bar        : 674

            Mcast pkts sent to the cross-bar        : 6660

            Ucast pkts received from the cross-bar  : 35265925

            Pkts sent to the port                   : 35275934

            Pkts discarded on ingress               : 0

            Per-priority-pause status               : Rx (Inactive), Tx (Inactive)

     

     

        qos-group 4

        q-size: 179200, q-size-40g: 200960, HW MTU: 9216 (9216 configured)

        drop-type: no-drop, xon: 62720, xoff: 88320

        Statistics:

            Pkts received over the port             : 1620866050

            Ucast pkts sent to the cross-bar        : 1338101557

            Mcast pkts sent to the cross-bar        : 192

            Ucast pkts received from the cross-bar  : 87178143

            Pkts sent to the port                   : 87178143

            Pkts discarded on ingress               : 298281129

            Per-priority-pause status               : Rx (Inactive), Tx (Inactive)

     

     

    To clear the counters use:

    switch (config) # clear qos statistics