0 Replies Latest reply on Jan 20, 2016 6:45 AM by jerome

    Mellanox OFED QoS and Rate-limit

    jerome

      Hi,

      I'm working on a testing environment where I want to test the QoS functionality in Mofed.

      I do think that I have a good comprehension of Basic InfiniBand QoS (SL and VL), and I try to understand Advanced QoS by defining policies (/etc/opensm/qos-policy.conf), and especially the rate-limit functionality.

       

      I basically have 2 groups (One EDR, one FDR) and I want to restrict the bandwidth when one node of one group is communicating with a node of the other group.

       

      Here is my qos-policy.conf :

       

      port-groups

          port-group

              name: nodeFDR

              port-guid: 0x0002c9030032ac41, 0x0002c9030032ac70, 0x0002c9030032acb0, 0x0002c9030032acc0, 0x0002c9030032af00

          end-port-group

          port-group

              name: nodeEDR

              port-guid: 0xe41d2d0300f29d70, 0xe41d2d0300f29eac, 0xe41d2d0300f29fb4

          end-port-group

      end-port-groups

      qos-levels

           qos-level

                   name: DEFAULT

                   use: default QoS Level

                   sl: 5

               rate-limit: 16

               end-qos-level

               qos-level

                   name: SDR1X

                   mtu-limit: 4

                   rate-limit: 2

               sl: 2

               end-qos-level

               qos-level

                   name: QDR4X

                   mtu-limit: 4

                   rate-limit: 6

                   sl: 4

               end-qos-level

      end-qos-levels

      qos-match-rules

          qos-match-rule

              source: nodeEDR

              destination: nodeFDR

              qos-level-name: SDR1X

          end-qos-match-rule

          qos-match-rule

              source: nodeFDR

              destination: nodeEDR

              qos-level-name: QDR4X

          end-qos-match-rule

      end-qos-match-rules

       

      So far so good, opensm seems to be okay with all that. Also When I do an saquery pathrecord, everything seems to be ok (I'm using ibtool from python-rdma)  :

       

      ibtool saquery PathRecord --sgid=fe80::e41d:2d03:f2:9fb4  --dgid=fe80::2:c903:32:ac41

      PathRecord dump:

              service_id...............0x0000000000000000

              dgid.....................fe80::2:c903:32:ac41

              sgid.....................fe80::e41d:2d03:f2:9fb4

              dlid.....................6

              slid.....................15

              RawTraffic...............0

              Reserved_353.............0

              FlowLabel................0

              HopLimit.................0

              tclass...................0x00

              Reversible...............1

              NumbPath.................0

              pkey.....................0xffff

              qos_class................0x000

              sl.......................2

              MTUSelector..............2

              mtu......................4

             RateSelector.............2

              rate.....................2

              PacketLifeTimeSelector...2

              pkt_life.................18

              preference...............0

              ReversePathPKeyMemberBit.0

              Reserved_466.............0

              Reserved_480.............0

       

      As expected, I don't have the same result for the reverse path :

       

      ibtool saquery PathRecord --dgid=fe80::e41d:2d03:f2:9fb4  --sgid=fe80::2:c903:32:ac41

      PathRecord dump:

              service_id...............0x0000000000000000

              dgid.....................fe80::e41d:2d03:f2:9fb4

              sgid.....................fe80::2:c903:32:ac41

              dlid.....................15

              slid.....................6

              RawTraffic...............0

              Reserved_353.............0

              FlowLabel................0

              HopLimit.................0

              tclass...................0x00

              Reversible...............1

              NumbPath.................0

              pkey.....................0xffff

              qos_class................0x000

              sl.......................4

              MTUSelector..............2

              mtu......................4

              RateSelector.............2

              rate.....................6

              PacketLifeTimeSelector...2

              pkt_life.................18

              preference...............0

              ReversePathPKeyMemberBit.0

              Reserved_466.............0

              Reserved_480.............0

       

      Starting from here, I'm running benchmarks (both ib_send_bw and ib_write_bw) but I'm not speed limited. I'm running wirespeed.

       

      Am I missing something here ?

       

      Thank you,

       

      Jérôme