3 Replies Latest reply on Nov 28, 2017 5:48 AM by inbusiness

    ESXi 6.5U1 40,56Gb Ethernet Performance problems



      I'm test lossy TCP/IP test on ESXi 6.5U1.


      Here is a sample link.

      Speed testing 40G Ethernet in the Homelab | Erik Bussink


      I'm also test with same ConnectX-3 MCX354A-FCBT with latest firmware.

      All bunch of MCX354A-FCBT with vSwitch MTU 9000 connect to SX6036G Gateway Switch systems with port MTU 9000


      Q01. Any switch mode (include VPI-single-mode or Ethernet mode 40GbE port configuration) in SX6036G 40GbE port configuration shows me a terrible performance level.




      But If change port mode to 10GbE mode shows me a normal 10GbE ethernet performance.


      Do you have a  any solution?



      Q02. latest MCX354-FCBT firmware can't link-up properly with SX6036G's 56GbE port mode.


      Do you have any a solution?

        • Re: ESXi 6.5U1 40GbE Performance problems

          Assuming you are using Mellanox-Native Esxi driver then as far as I recall - Esxi6.5 does not support Connectx-3 adapter (only Cx-4/5)

          Nevertheless, from your description, it looks though your are running a VM to VM test

          I would suggest that you first "dismount" the vswitch / virtual adapter mode and try a simple "bare-metal" 40G performance test from an Esxi_server-to-Esxi_server, meaning from a physical adapter (CX-3) to another CX-3 adapter
          Ensure you have identical MTU size on both switch & Exsi server (MTU 9000)

          Also suggesting you approach Mellanox website to read more about the Esxi6.5 va. Mellanox adapter compatibility matrix


            • Re: ESXi 6.5U1 40GbE Performance problems

              I test seperate VM on local SSD & ESXi 6.5 host.


              Here is briefs

              ESX host 01 - iPerf VM 01 - SX6036G - ESX host 02 - iPerf VM 02


              All switch & vSwitch, vKernel adapter, nmlx_en adapter set to 9k MTU.


              And ESXi 6.5 update 1 inbox driver include nmlx4_core that support Mellanox ConnectX-3, Pro and belows.

              * https://www.google.co.kr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwiIzs2Ozo_WAhXFxLwKHbYvBswQFg…


              Why did you comment that based your personal memorandum?


              In another thread on this community said CX-3 supported with ESXi 6.5 inbox driver.


              I'll test Arista 40Gb switch with same configuration then I'll update result.


              Best Regard,

              Jae-Hoon Choi


              Update 01. Direct connection between 2 of CX-3s test

              This test shows almost 19Gb/s grade performance



              Update 02. SX6036G Global Pause On switched test

              This test also shows almost 19Gb/s grade performance


              Update 03. SX6036G all port 10GbE switched test

              This test also shows almost 10Gb/s grade performance


              Update 04. ESXi inbox Ethernet driver packet drop bug - ESXi host iPerf test results...:(

              It must have a packet drop & error bug in Mellanox inbox ESXi drivers.

              How do you think about below?



              Update 05. Windows, Linux Host test

              All Windows, Linux Host show me a greate performance on 40GbE configurations that include CX-3 HCAs & SX6036G switches.


              Best Regard,

              Jae-Hoon Choi


              Update 01.

              I can build a 56Gb Ethernet TCP/IP network with ConnectX-3 (firmware 2.42.5000) & SX6036G (firmware 3.6.4006), now.

              packet drop was not appeared that I don't use PFC configuration with ConnectX-3 ESXi 6.5 update 1.


              40, 56GbE Link-up require small delay to establish requested speed.

              But performance level is terrible like below.


              In 56GbE mode, I'm test iPerf test between physical ESXi host then I met a almost 6.11Gb/s performance only.


              * 56GbE iPerf client - physical ESXi 6.5 update 1 host 01 with MTU 9000

              * 56GbE iPerf server - physical ESXi 6.5 update 1 host 02 with MTU 9000

              Is there any configuration shold be need on SX6036G?


              * This is a traditional TCP/IP Network, not RDMA Network.

              * When 56Gb ethernet show me a 56GbE throughput, I'll switch to another test like iSER driver and so on.



              Jae-Hoon Choi

                • Re: ESXi 6.5U1 40,56Gb Ethernet Performance problems

                  I found what's the problem myself.


                  All of 40GbE or above architecture on the planet consists of 4 lane configuration.

                  That's a 1st problem, 2nd is Mellanox CX-3 chipset's TSO bugs.

                  If you disable TSO offload then you'll meet a improve of performance, but also meet a high CPU usages.


                  I gave up the performance TCP/IP tuning with 40GbE or above.



                  Jae-Hoon Choi