First of all I'd just like to say I think its excellent that Mellanox provides a forum for IB home labs / hobby, that is very good service.
For me, my home lab is a way of testing out "new" solutions before I consider recommending them - I like to have full confidence in my recommendations and of course its a great way to push your skills where you may not be able to do in a corporate / budget environment.
Anyway, I am very new to IB but with it becoming prominent in the Big Data market and also VSANs (storage in the cabinet); I wanted to see what it was all about.
I have purchased the following (I reliase its not cutting edge, but early next year I will be upgrading to a QDR / 40Gbps Mellanox IB switch with built-in SM) assuming I can get this working the way I expect:
1 x Voltaire GridDirector ISR 9024D (not the M model)
2 x MHGH28-XTC (Rev X1) HCA cards - I flashed these to firmware version 2.7000
2 x CX4 cables
2 x VMware ESXi custom systems
2 x Intel 335 SSDs (500MB/s each) - in 2 weeks this will become 4 x Intel 335 SSDs (providing theoretical 2Gbps ish IO in RAID-0)
Ok, I have installed the relevant drivers - for sake of a simple guide (which can be corrected if you think I have missed / done something wrong) here is what I did:
[ INFINIBAND ]
1. Install the Mellanox OFED drivers
esxcli system module paramters set -m=mlx4_core -p=mtu_4k=1
esxcli software vib install -d /tmp/mlx4_en-mlnx-126.96.36.199-offline_bundle-471530.zip –no-sig-check
esxcli software vib install -d /tmp/MLNX-OFED-ESX-188.8.131.52.zip
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: Mellanox_bootbank_net-ib-cm_184.108.40.206-1OEM.500.0.0.472560, Mellanox_bootbank_net-ib-core_220.127.116.11-1OEM.500.0.0.472560, Mellanox_bootbank_net-ib-ipoib_18.104.22.168-1OEM.500.0.0.472560, Mellanox_bootbank_net-ib-mad_22.214.171.124-1OEM.500.0.0.472560, Mellanox_bootbank_net-ib-sa_126.96.36.199-1OEM.500.0.0.472560, Mellanox_bootbank_net-ib-umad_188.8.131.52-1OEM.500.0.0.472560, Mellanox_bootbank_net-mlx4-core_184.108.40.206-1OEM.500.0.0.472560, Mellanox_bootbank_net-mlx4-ib_220.127.116.11-1OEM.500.0.0.472560, Mellanox_bootbank_scsi-ib-srp_18.104.22.168-1OEM.500.0.0.472560
esxcli software acceptance set --level=CommunitySupported
esxcli software vib install -v /tmp/ib-opensm-3.3.16.x86_64.vib --no-sig-check
3. Fix MTU and partitions.conf
cp partitions.conf /scratch/opensm/0x001a4bffff0c1399/
cp partitions.conf /scratch/opensm/0x001a4bffff0c139a/
4. Flashed both HCA cards to firmware 2.7000
5.Created a virtual network in ESXi using one port on the HCA each (per ESXi system) - ESXi recognises this vnic as up and 20Gbps
6. TRIED to set the MTU > 2k but failed, won't go higher than 2k in the vswitch.
7. Created 2 x WIN7 systems each with 2x4GHz vCPUs, 8GB RAM, 1 x SSD based HDD (theoretical 500MB/s or slightly less IO - no other VM using this SSD datastore) and configured NICs using IP on the IPoIB vswitch same subnet, ping works etc
8. Copied a 3.6GB ISO from WIN701 to WIN702 - 289Mbps (15secs) - thats fast but I was expecting more throughput
9. Created a 4GB RAM disk on each system
10. Re-copied the above file, result: 360MB/sec
I was expecting much quicker than this copy rates, especially via RAMdisk - are there any areas that you can suggest I look at as this is not performing at the level I'd expect.