0 Replies Latest reply on Dec 6, 2018 5:17 PM by pattim

    Opensubnetmanager (openSM) for a switchless setup

    pattim

      Hello:  I have three 12-core server boxes directly connected with ConnectX-2 Mellanox cards.  The head node has a dual port MNPH29C-XTR card with each port connected to a separate compute node (direct connect; no IB switch).  Each compute node has a single-port MNP* card.  The two ports on the head node are configured with as separate subnets, which I believe is the correct way to set up a multiport fabric, and all hardware seems to be working correctly.  I can't seem to get OpenSM (the subnet manager) to "control/map" both ports (subnets) on the head node at the same time.  I need this so that OpenMPI can use both compute nodes in one calculation process.  I have OpenSM running and it detects the hardware, but claims it can only work with one subnet at a time (as I understand OpenMPI, the first compute node needs to be able to access both the head node and the other compute node for OpenMPI to work). 

       

      I can carry out a calculation using head + first compute node or else head + second compute node, but I can't seem to get head + first + second nodes all participating in a calculation.  Is it wrong to have both ports on the head node defined to separate subnets?

       

      Can anyone help me understand how to configure this kind of setup?  I *thought* this was the purpose of OpenSM - i.e., to map all nodes and to thereby bypass the need for a hardware IB switch.

       

      Thank You Very Much!!!!!   (hoping for help)

      PattiMichelle