4 Replies Latest reply on Dec 14, 2017 12:45 PM by gxs120

    Configuring inter-switch links between SX6036 QSFP ports

    gxs120

      Apologies in advance as I am new to posting.

      I am responsible for configuring 2 SX6036 switches.  Another team is responsible for configuring the server and Infiniband adapters on the servers.

      This is the network topology:

      IBM Knowledge Center

      The only deviation is there are 3 member servers, not 4 and each member is patched to the IB switches in the same manner as the CF servers (2 adapters each with 2 ports) .

      The subnet manager is running on both SX6036 switches.  HA is running over the switch management ports with one switch set with a higher priority (master) and the other is standby.  My understanding is that there is no other configuration required on the switches.

       

      1) How are the inter-switch links used?  My understanding is that they are only used if there is an adapter failure on a server.  True or False?

      2) If an adapter fails, how do the inter-switch links kick in?  Are they automatically configured to switch any IB traffic on any IB subnet?  The servers are configured with IP over IB.

      3) The diagram refers to the inter-switch links being setup as a LACP channel.  Is this done automatically or is there some configuration I need to configure?

       

      Thanks in advance for any help!

       

      Greg

        • Re: Configuring inter-switch links between SX6036 QSFP ports
          yairi

          Hi Greg,

           

          Infiniband architecture is a bit different than Ethernet in a way where the protocol uses all the ISL connections all together. In short, the Subnet Manager (SM), calculate and provision static routes between every end point (HCA port) to every other end point. This is typically done using a set of rules that are relative to the topology and are configuring those routes while avoiding the risk of having a loop in the network. The configuration also aims to spread the routes as evenly as possible across all the available links to maximize the BW utilization within the fabric itself.

          If a situation occurs where a link goes down during the life-cycle of the network, the SM has the way to identify the event (usually receives a trap) and recalculate the routes again using alternative routes. This is all done automatically - no need to configure anything.

          Lastly, the notion of "LACP" does not exist in Infiniband - it is just not needed.

           

          I hope this helps. You can read more about the Infiniband architecture online. There are many resources at different levels.

           

          Cheers!