3 Replies Latest reply on Jul 20, 2018 2:11 PM by stepheny

    Can't get full FDR bandwidth with Connect-IB card in PCI 2.0 x16 slot

    ewalter

      Greetings,

       

      We have some nodes (Dell R415s running RHEL 6.8) with Connect-IB cards in a PCI 2.0 x16 slot (the only one available) and can't seem to get more than 45 Gbit/s using ib_send_bw.  I have two of the nodes connected directly using a new FDR cable and with the SM running on one of the nodes.  I have updated the BIOS, the OFED and the HCA firmware on both nodes.  Still I can't seem to get the full FDR bandwidth.  The connect-ib product page (http://www.mellanox.com/page/products_dyn?product_family=142&mtag=connect_ib ) states the following:

       

      "Connect-IB also enables PCI Express 2.0 x16 systems to take full advantage of FDR, delivering at least twice the bandwidth of existing PCIe 2.0 solutions."

       

      Since the PCIe 2.0 x16 is able to support 64 Gbit/s in one direction, shouldn't I be able to achieve full FDR (~54ish Gbit/s) as the product page implies.  Or, am I wrong and there is some extra overhead that reduces the bandwidth for PCIe 2.0 x16 vs. PCIe 3.0 x16?

       

      I have gone through the Performance Tuning for Mellanox Adapters guide and there isn't much more that I can try based on this.  The latest BIOS has nowhere near the number of setting that are suggested to be tweaked in the guide.  I have also tried mlnx_tune and get one warning:

      ----------------------------------------------------------

      Connect-IB Device Status on PCI 01:00.0

      FW version 10.16.1200

      OK: PCI Width x16

      Warning: PCI Speed 5GT/s >>> PCI width status is below PCI capabilities. Check PCI configuration in BIOS.                       <--------------

      PCI Max Payload Size 128

      PCI Max Read Request 512

      Local CPUs list [0, 1, 2, 3, 4, 5]

      ----------------------------------------------------------

      But this is probably correct since I am using a PCIe 2.0 x16 slot (PCIe 2.0 can only do 5 GT/s), right?

       

      Here is the output of ibv_devinfo:

      -------------------------------------------

      hca_id: mlx5_0

      transport: InfiniBand (0)

      fw_ver: 10.16.1200

      node_guid: f452:1403:002e:eb40

      sys_image_guid: f452:1403:002e:eb40

      vendor_id: 0x02c9

      vendor_part_id: 4113

      hw_ver: 0x0

      board_id: MT_1220110019

      phys_port_cnt: 1

      Device ports:

      port: 1

      state: PORT_ACTIVE (4)

      max_mtu: 4096 (5)

      active_mtu: 4096 (5)

      sm_lid: 1

      port_lid: 2

      port_lmc: 0x00

      link_layer: InfiniBand

      -------------------------------------------

      and iblinkinfo:

      -------------------------------------------

      CA: A HCA-1:

            0xf4521403002ee9f0      1    1[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       2    1[  ] "B" ( )

      CA: B HCA-1:

            0xf4521403002eeb40      2    1[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       1    1[  ] "A" ( )

      -------------------------------------------

       

      Can anyone tell me if this is the best I can expect or is there something else I can change to achieve FDR bandwidth with these HCAs?

       

      Thanks in advance!

       

      Eric

        • Re: Can't get full FDR bandwidth with Connect-IB card in PCI 2.0 x16 slot
          stepheny

          Server:
          > ib_send_bw -a -F --report_gbits
          Client:
          > ib_send_bw -a -F --report_gbits <serverIP>

           

          Please let me know your results and thank you...

          ~Steve

            • Re: Can't get full FDR bandwidth with Connect-IB card in PCI 2.0 x16 slot
              ewalter

              Hi Steve,

               

              Here you are.  Thanks for taking an interest.  Still getting ~45 Gb/sec on both client and server.  Here is the client output:

               

              [root@vx01 ~]#  ib_send_bw -a -F --report_gbits vx02

              ---------------------------------------------------------------------------------------

                                  Send BW Test

              Dual-port       : OFF Device         : mlx5_0

              Number of qps   : 1 Transport type : IB

              Connection type : RC Using SRQ      : OFF

              TX depth        : 128

              CQ Moderation   : 100

              Mtu             : 4096[B]

              Link type       : IB

              Max inline data : 0[B]

              rdma_cm QPs : OFF

              Data ex. method : Ethernet

              ---------------------------------------------------------------------------------------

              local address: LID 0x3e4 QPN 0x005e PSN 0x239861

              remote address: LID 0x3e6 QPN 0x004c PSN 0xed4513

              ---------------------------------------------------------------------------------------

              #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]

              2          1000           0.098220            0.091887            5.742968

              4          1000             0.20               0.19       6.037776

              8          1000             0.40               0.39       6.071169

              16         1000             0.78               0.67       5.220818

              32         1000             1.53               1.43       5.576730

              64         1000             3.16               3.10       6.053410

              128        1000             6.20               6.16       6.012284

              256        1000             12.35              12.28     5.997002

              512        1000             22.67              22.47     5.486812

              1024       1000             38.02              36.69     4.478158

              2048       1000             42.26              42.04     2.565771

              4096       1000             43.82              43.68     1.332978

              8192       1000             44.63              44.63     0.681005

              16384      1000             44.79              44.79     0.341728

              32768      1000             45.21              45.21     0.172449

              65536      1000             45.35              45.35     0.086506

              131072     1000             45.45              45.45     0.043342

              262144     1000             45.45              45.45     0.021670

              524288     1000             45.47              45.47     0.010840

              1048576    1000             45.47              45.47     0.005421

              2097152    1000             45.48              45.48     0.002711

              4194304    1000             45.48              45.48     0.001355

              8388608    1000             45.48              45.48     0.000678

              ---------------------------------------------------------------------------------------

               

              Here is the server output:

               

              [root@vx02 ~]#  ib_send_bw -a -F --report_gbits

               

               

              ************************************

              * Waiting for client to connect... *

              ************************************

              ---------------------------------------------------------------------------------------

                                  Send BW Test

              Dual-port       : OFF Device         : mlx5_0

              Number of qps   : 1 Transport type : IB

              Connection type : RC Using SRQ      : OFF

              RX depth        : 512

              CQ Moderation   : 100

              Mtu             : 4096[B]

              Link type       : IB

              Max inline data : 0[B]

              rdma_cm QPs : OFF

              Data ex. method : Ethernet

              ---------------------------------------------------------------------------------------

              local address: LID 0x3e6 QPN 0x004c PSN 0xed4513

              remote address: LID 0x3e4 QPN 0x005e PSN 0x239861

              ---------------------------------------------------------------------------------------

              #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]

              2          1000           0.000000            0.099141            6.196311

              4          1000             0.00               0.20       6.229974

              8          1000             0.00               0.40       6.265230

              16         1000             0.00               0.69       5.362016

              32         1000             0.00               1.47       5.727960

              64         1000             0.00               3.22       6.283794

              128        1000             0.00               6.34       6.191118

              256        1000             0.00               12.64     6.169975

              512        1000             0.00               23.08     5.634221

              1024       1000             0.00               37.53     4.581582

              2048       1000             0.00               42.63     2.602155

              4096       1000             0.00               44.07     1.344970

              8192       1000             0.00               45.04     0.687191

              16384      1000             0.00               45.04     0.343602

              32768      1000             0.00               45.35     0.172994

              65536      1000             0.00               45.45     0.086690

              131072     1000             0.00               45.52     0.043409

              262144     1000             0.00               45.51     0.021699

              524288     1000             0.00               45.52     0.010852

              1048576    1000             0.00               45.52     0.005427

              2097152    1000             0.00               45.53     0.002714

              4194304    1000             0.00               45.53     0.001357

              8388608    1000             0.00               45.53     0.000678

              ---------------------------------------------------------------------------------------

               

              Please let me know if you want any other info and I will send it straight away.

               

              Regards,

               

              Eric