3 Replies Latest reply on Nov 28, 2017 5:48 AM by inbusiness

    ESXi 6.5U1 40,56Gb Ethernet Performance problems

    inbusiness

      Hi!

      I'm test lossy TCP/IP test on ESXi 6.5U1.

       

      Here is a sample link.

      Speed testing 40G Ethernet in the Homelab | Erik Bussink

       

      I'm also test with same ConnectX-3 MCX354A-FCBT with latest firmware.

      All bunch of MCX354A-FCBT with vSwitch MTU 9000 connect to SX6036G Gateway Switch systems with port MTU 9000

       

      Q01. Any switch mode (include VPI-single-mode or Ethernet mode 40GbE port configuration) in SX6036G 40GbE port configuration shows me a terrible performance level.

      40GbE_ESXi6.5U1_test.png

       

       

      But If change port mode to 10GbE mode shows me a normal 10GbE ethernet performance.

       

      Do you have a  any solution?

       

       

      Q02. latest MCX354-FCBT firmware can't link-up properly with SX6036G's 56GbE port mode.

       

      Do you have any a solution?

        • Re: ESXi 6.5U1 40GbE Performance problems
          aviap

          Assuming you are using Mellanox-Native Esxi driver then as far as I recall - Esxi6.5 does not support Connectx-3 adapter (only Cx-4/5)

          Nevertheless, from your description, it looks though your are running a VM to VM test

          I would suggest that you first "dismount" the vswitch / virtual adapter mode and try a simple "bare-metal" 40G performance test from an Esxi_server-to-Esxi_server, meaning from a physical adapter (CX-3) to another CX-3 adapter
          Ensure you have identical MTU size on both switch & Exsi server (MTU 9000)

          Also suggesting you approach Mellanox website to read more about the Esxi6.5 va. Mellanox adapter compatibility matrix
          http://www.mellanox.com/page/products_dyn?product_family=29

           

            • Re: ESXi 6.5U1 40GbE Performance problems
              inbusiness

              I test seperate VM on local SSD & ESXi 6.5 host.

               

              Here is briefs

              ESX host 01 - iPerf VM 01 - SX6036G - ESX host 02 - iPerf VM 02

               

              All switch & vSwitch, vKernel adapter, nmlx_en adapter set to 9k MTU.

               

              And ESXi 6.5 update 1 inbox driver include nmlx4_core that support Mellanox ConnectX-3, Pro and belows.

              * https://www.google.co.kr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwiIzs2Ozo_WAhXFxLwKHbYvBswQFg…

               

              Why did you comment that based your personal memorandum?

               

              In another thread on this community said CX-3 supported with ESXi 6.5 inbox driver.

               

              I'll test Arista 40Gb switch with same configuration then I'll update result.

               

              Best Regard,

              Jae-Hoon Choi

               

              Update 01. Direct connection between 2 of CX-3s test

              This test shows almost 19Gb/s grade performance

               

               

              Update 02. SX6036G Global Pause On switched test

              This test also shows almost 19Gb/s grade performance

               

              Update 03. SX6036G all port 10GbE switched test

              This test also shows almost 10Gb/s grade performance

               

              Update 04. ESXi inbox Ethernet driver packet drop bug - ESXi host iPerf test results...:(

              It must have a packet drop & error bug in Mellanox inbox ESXi drivers.

              How do you think about below?

               

               

              Update 05. Windows, Linux Host test

              All Windows, Linux Host show me a greate performance on 40GbE configurations that include CX-3 HCAs & SX6036G switches.

               

              Best Regard,

              Jae-Hoon Choi

               

              Update 01.

              I can build a 56Gb Ethernet TCP/IP network with ConnectX-3 (firmware 2.42.5000) & SX6036G (firmware 3.6.4006), now.

              packet drop was not appeared that I don't use PFC configuration with ConnectX-3 ESXi 6.5 update 1.

               

              40, 56GbE Link-up require small delay to establish requested speed.

              But performance level is terrible like below.

               

              In 56GbE mode, I'm test iPerf test between physical ESXi host then I met a almost 6.11Gb/s performance only.

               

              * 56GbE iPerf client - physical ESXi 6.5 update 1 host 01 with MTU 9000

              * 56GbE iPerf server - physical ESXi 6.5 update 1 host 02 with MTU 9000

              Is there any configuration shold be need on SX6036G?

               

              * This is a traditional TCP/IP Network, not RDMA Network.

              * When 56Gb ethernet show me a 56GbE throughput, I'll switch to another test like iSER driver 1.0.0.1 and so on.

               

              BR,

              Jae-Hoon Choi

                • Re: ESXi 6.5U1 40,56Gb Ethernet Performance problems
                  inbusiness

                  I found what's the problem myself.

                   

                  All of 40GbE or above architecture on the planet consists of 4 lane configuration.

                  That's a 1st problem, 2nd is Mellanox CX-3 chipset's TSO bugs.

                  If you disable TSO offload then you'll meet a improve of performance, but also meet a high CPU usages.

                   

                  I gave up the performance TCP/IP tuning with 40GbE or above.

                   

                  Regards,

                  Jae-Hoon Choi