Can you try to run ib bandwidth test ?
ib_write_bw / ib_read_bw
Do you run on your bare metal or on a VM ?
its ethernet not infiniband
its bare metal esxi 6.5
how do i test it ?
Did you go through the Tuning guide
Performance Tuning for Mellanox Adapters
Did you try to run the iperf with taskset to specify the cores that are on the same NUMA of the adapter ?
List your numa node
mst status -v
lscpu will show you the CPU's on your NUMA node (like example)
NUMA node0 CPU(s): 0-3
NUMA node1 CPU(s): 4-7
Did you manage to get better performance ?
Not yet done. Ive read through the guide, but didnt find a compatible package yet for VMWare.
Is there a package for VMWare ESXI 6.5 which has the tools like raw_ethernet_bw included ?
ive updated my boxes to the latest Connect-X 4 drivers from mellanox but i cant seem to execute the commands you listed before
For mst command:
- 1. Download the MFT for VMware vib package from: http://www.mellanox.com/products/management_tools.php
- 2. Install the package. Run:
# esxcli software vib install -v MFT
# esxcli software vib install -v MST
NOTE: For VIBs installation examples, please see below.
- 3. Reboot system.
4. Start the mst driver. Run:
# /opt/mellanox/bin/mst start
For performance see
installed and rebootet:
:/opt/mellanox/bin] ./mst status -v
DEVICE_TYPE MST PCI RDMA NET NUMA
ConnectX4(rev:0) mt4115_pciconf0 88:00.0 net-vmnic6
ConnectX4(rev:0) mt4115_pciconf0.1 88:00.1 net-vmnic7
Querying Mellanox devices firmware ...
Device Type: ConnectX4
Part Number: MCX416A-CCA_Ax
Description: ConnectX-4 EN network interface card; 100GbE dual-port QSFP28; PCIe3.0 x16; ROHS R6
PCI Device Name: mt4115_pciconf0
Versions: Current Available
FW 12.16.1020 N/A
PXE 3.4.0812 N/A
I think its all automated in vmware esxi 6.5 now ?
Constrains the set of NUMA nodes on which a virtual machine's virtual CPU and memory can be scheduled.Note:
When you constrain NUMA node affinities, you might interfere with the ability of the NUMA scheduler to rebalance virtual machines across NUMA nodes for fairness. Specify NUMA node affinity only after you consider the rebalancing issues.