Assuming you are using Mellanox-Native Esxi driver then as far as I recall - Esxi6.5 does not support Connectx-3 adapter (only Cx-4/5)
Nevertheless, from your description, it looks though your are running a VM to VM test
I would suggest that you first "dismount" the vswitch / virtual adapter mode and try a simple "bare-metal" 40G performance test from an Esxi_server-to-Esxi_server, meaning from a physical adapter (CX-3) to another CX-3 adapter
Ensure you have identical MTU size on both switch & Exsi server (MTU 9000)
Also suggesting you approach Mellanox website to read more about the Esxi6.5 va. Mellanox adapter compatibility matrix
I test seperate VM on local SSD & ESXi 6.5 host.
Here is briefs
ESX host 01 - iPerf VM 01 - SX6036G - ESX host 02 - iPerf VM 02
All switch & vSwitch, vKernel adapter, nmlx_en adapter set to 9k MTU.
And ESXi 6.5 update 1 inbox driver include nmlx4_core that support Mellanox ConnectX-3, Pro and belows.
Why did you comment that based your personal memorandum?
In another thread on this community said CX-3 supported with ESXi 6.5 inbox driver.
I'll test Arista 40Gb switch with same configuration then I'll update result.
Update 01. Direct connection between 2 of CX-3s test
This test shows almost 19Gb/s grade performance
Update 02. SX6036G Global Pause On switched test
This test also shows almost 19Gb/s grade performance
Update 03. SX6036G all port 10GbE switched test
This test also shows almost 10Gb/s grade performance
Update 04. ESXi inbox Ethernet driver packet drop bug - ESXi host iPerf test results...:(
It must have a packet drop & error bug in Mellanox inbox ESXi drivers.
How do you think about below?
Update 05. Windows, Linux Host test
All Windows, Linux Host show me a greate performance on 40GbE configurations that include CX-3 HCAs & SX6036G switches.
I can build a 56Gb Ethernet TCP/IP network with ConnectX-3 (firmware 2.42.5000) & SX6036G (firmware 3.6.4006), now.
packet drop was not appeared that I don't use PFC configuration with ConnectX-3 ESXi 6.5 update 1.
40, 56GbE Link-up require small delay to establish requested speed.
But performance level is terrible like below.
In 56GbE mode, I'm test iPerf test between physical ESXi host then I met a almost 6.11Gb/s performance only.
* 56GbE iPerf client - physical ESXi 6.5 update 1 host 01 with MTU 9000
* 56GbE iPerf server - physical ESXi 6.5 update 1 host 02 with MTU 9000
Is there any configuration shold be need on SX6036G?
* This is a traditional TCP/IP Network, not RDMA Network.
* When 56Gb ethernet show me a 56GbE throughput, I'll switch to another test like iSER driver 126.96.36.199 and so on.
I found what's the problem myself.
All of 40GbE or above architecture on the planet consists of 4 lane configuration.
That's a 1st problem, 2nd is Mellanox CX-3 chipset's TSO bugs.
If you disable TSO offload then you'll meet a improve of performance, but also meet a high CPU usages.
I gave up the performance TCP/IP tuning with 40GbE or above.