3 Replies Latest reply on Jul 27, 2017 5:48 PM by bgad15

    Windows ConnectX-3 High Latency


      I am having trouble with latency on a Windows Server IB setup.  I have read and tried many troubleshooting steps on various forums and the Mellanox Performance Tuning Guide but have had no luck in getting my setup to the low latency that I imagine the hardware is capable of.  I am a complete beginner in the IB space so it is very possible I am missing some very basic config or concept.


      I guess it is also possible that 0.2ms to 0.4ms ping latency and 2.5ms additional file access latency is to be expected, please let me know if my expectations were off.


      Any advice would be greatly appreciated.


      ==== Hardware/Software Setup ====


          Adapter: MCX354A-FCBT

          Firmware: 2.40.7000

          Motherboard: SuperMicro X8DT3

          PCIe Slot: Gen2 8x (No Gen3 available)

          OS: Windows Server 2016 DC

          Mellanox Port Mode: IB




          Adapter: MCX354A-FCBT

          Firmware: 2.40.7000

          Motherboard: SuperMicro X8DTN+

          PCIe Slot: Gen2 8x (No Gen3 available)

          OS: Windows Server 2016 DC

          Mellanox Port Mode: IB




          Direct Connect (Back to Back): Mellanox MC2207130-002


      ==== VSTAT ====

      Here are the vstat outputs for each host.


      HOST01:             COMMAND: vstat.exe

                                  RESULT: https://pastebin.com/raw/aCk0Ltp4


      HOST02:             COMMAND: vstat.exe

                                  RESULT: https://pastebin.com/raw/L0zWr7jk


      ==== Ping Times ====

      The ping times seem very high (0.32ms to 0.45ms)  In fact they are the same as my 1GB Ethernet connection.


      HOST01:             COMMAND: hrping.exe

                                  RESULT: https://pastebin.com/raw/ubV0ZnsF


      HOST02:             COMMAND: hrping.exe

                                  RESULT: https://pastebin.com/raw/QVTYKkXQ


      **Note: ibping produces slightly better times of 0.19ms to 0.24ms




      ==== Throughput ====

      The throughput seems "fine" I guess.  I read various sources that said 56Gbps is limited to lower real world throughput for various reasons.  In any case, I am not too concerned with throughput since I am focused on IOPS.



      HOST01:             COMMAND: ntttcp.exe -r -m 28,*, -rb 2M -a 16 -t 5

                                  RESULT:  https://pastebin.com/raw/RmQSBL2G


      HOST02:             COMMAND: ntttcp.exe -s -m 28,*, -l 512K -a 2 -t 5

                                  RESULT:  https://pastebin.com/raw/djsVFs8R




      ==== IOPS ====

      So, finally to my actual problem.  I have a disk on HOST01 that locally has ~90K IOPS with 0.3ms latency but over the network it is down to ~10K IOPS and up to 3.0ms+ latency.


      HOST01 (Local):     COMMAND: diskspd.exe -b8K -d30 -o4 -t8 -h -r -w0 -L -Z1G -c20G x:\share\iotest.dat

                                      RESULT: https://pastebin.com/raw/hikPjDQs


      HOST02 (Over IB):   COMMAND: diskspd -b8K -d30 -o4 -t8 -h -r -w0 -L -Z1G -c20G \\\shared\iotest.dat

                                       RESULT: https://pastebin.com/raw/e07Ajx1i      

        • Re: Windows ConnectX-3 High Latency

          Hi Sam,


          Initially I see two issues with your setup. Your PCIe slots and the configuration of your diskspd tests.


          Maximum throughput on a PCIe 2.0 x8 is ~32Gbit/s

          PCIe 3.0 x8 is roughly double at ~64Gbit/s - So you're PCIe 2.0 x8 slots are going to be the bottleneck when trying to push 40Gb/s let alone 56Gb/s speeds.


          As for latency, 0.3ms on the network is fine - I have similar latency as you.


          But for the diskspd tests you'll need to provide more information, are jumbo frames enabled? RDMA enabled? Also, are you writing to the same type of disk on both tests? It seems like it's more an issue with SMB configuration on the host than anything to do with the adaptors.


          If you want to have a look at adaptor performance, I'd recommend using a tool such as iPerf.

            • Re: Windows ConnectX-3 High Latency

              I didn't realize the bottleneck existed as far as throughput goes on gen2 vs gen3.  Thanks for pointing that out, but in this case I think I'm going to have to leave it since the servers are in place and upgrading motherboards is out of scope for this project at this point.  But good to know even more performance can be had.


              As for your questions and suggestions:


              1.)  Jumbo Frames are enabled at 4092 on each adapter.

              2.)  RDMA  is enabled and working - verified by using perfmon and adding RDMA counters.

              3.)  The disk is the same exact disk on both tests - just shared over SMB on the remote test and accessed via drive letter on the local test.

              4.)  I did new tests using iPerf but as far as I can tell it cannot test IOPS or latency.  I did my original tests with NTTTCP because that is what was recommended in various Mellanox documents when testing on Windows.  In any case, both iPerf and NTTTCP show the same results of just under 30Gbps which would seem to be the limit of the gen2 8x slots plus some overhead.



              On the latency subject - I am a bit confused - and it very well could be that I am just missing something.   You say that the 0.3ms latency is fine, but that is the same latency I have with my standard 1GBE nics.  I was under the impression that IB had much lower latency than standard 1GB Ethernet.   Is ping not the correct way to see the lower latency that IB features?