I purchased a bunch of Voltaire 400EX-D HCAs (dual-port) and installed them into IBM x3650 servers. I then installed Rocks 6.1.1 (The ClusterIQ version based on Centos 6.5). The MLNX_OFED version that comes bundled with this version of Rocks has MLNX_OFED version 2.x and did not work with my HCAs. I then had to rebuild MLNX_OFED 1.5.3-4.042 to allow support for the Centos 6.5 kernel. My cluster now works OK and I am able to compile and run MPI jobs on OpenFoam and a varietz of other applications.
However, I am unable to use the MXM (1.5.8) functionality and had to disable knem in /usr/mpi/gcc/openmpi/1.6.4/etc/openmpi-mca-params.conf to prevent error messages and jobs failing.
Do I need to install an older version of MXM and FCA, and if so, how do I do that while ensuring support for the CentOS 6.5 kernel?