2 Replies Latest reply on Sep 28, 2015 11:53 AM by realist

    IS5022 Switch Fabric

      Hello,

       

      I'm very new to infiniband, and looking to reuse some gear that already exists in my organization.

       

      I have 6 servers with dual single port infiniband cards (MHQH19B-XTR), and currently have 1 IS5022 switch. The servers are running IPoIB on CentOS 7 and only using infiniband for IP traffic.

       

      I'd like to remove the switch as a single point of failure, and my first thought was to add a 2nd switch and 'stack' them as I would in Ethernet land. From the product overviews on the IS5022 unmanaged switch it looks like its possible to connect another IS5022 switch and connect the 2nd ib adapter of each server, as well as a connection between both IS5022 switches 'out of the box'.

       

      I would then bond the ib0 and ib1 interfaces on each server, configured in mode 1 (active/backup).

       

      Is this possible 'out of the box'? At a basic level it looks as though building a mesh topology is one of the primary purposes of infiniband. If so, then my greatest concern is bonding IPoIB.

       

      This is how I see the physical topology (small boxes on each device are ports):

       

      infiniband-network-topology-HA.png

       

      Thanks,

      Steve

        • Re: IS5022 Switch Fabric

          IS5022 is externally managed. I think in your case at least one switch needs to be manageable. But I might be wrong.

            • Re: IS5022 Switch Fabric
              realist

              Hi Steve,

              The topology looks fine.Since you have IS5022 switches they are "unmanaged" and are not capable of running an SM (subnet manager) so the other step you would need to take is to start opensm on 1 or 2 (if back SM is wanted) of your hosts compute1/2.The SM is responsible for discovering all IB (Infiniband) devices and assisgning a LID to each then setting up all the possible routing paths to each and every deivce and propogating the routes to each swtich in the fabric.Hope this helps.