1 of 1 people found this helpful
i am not familiar with the specific model of the server you mentioned but generally speaking this is what you need to get done for gaining the most out of your card:
- install the card on a PCI-e Gen-3 capable PCI slot. choose the one that supports x8 or x16 speeds. Make sure gen-3 is enabled in BIOS (i've seen gen-3 systems that were shipped with factory defaults supporting gen-2)
- make sure your card runs the latest firmware
- There are few protocols that runs on top of IB. some utilize the transport efficiently; some other less. if you want to see wire-speed rates, shoot for using some native IB protocol like RDMA or other protocol that natively speaks with the IB verbs layer - this is a KEY part.since you didn't mention what you application is like, i can't really comment further.
That's it - start with the above and see how far you can go. let me know..
If latency is important to you then NUMA affinity really makes a difference. If you run your test (i.e. ib_write_lat) in the same socket where the FDR card is connected, you can achieve latencies under 1 usec. Running in another socket will be ~20% slower. This does not affect the throughput, not in a noticeable way anyways.
IRQ affinity is also very important. BIOS setting too.
There is a very nice, and relatively short, Tuning Guide published by Mellanox which I think is a must: http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf