Recommendation for Inter-Switch Cabling

Version 5

    There are specific physical requirements applied when you cable Clos and tree-like topologies together. One of them specifies that data will (or must) be routed using MinHop or Up/Down routing algorithms. If you disregard these requirements it can result in unintended blocking conditions. For existing cabling or situations that force ‘bad’ cabling, we recommend a workaround that involves using Subnet Management configuration files.

    This post covers details of connecting InfiniBand switches using multiple parallel cables.

     

    References

     

    Inter-Switch Cabling

    Requirement: When you have a set of N links that connect two adjacent switches, you must ensure that for each switch the N cables occupy contiguous port numbers.

    Refer to the following diagram.  It is simply a set of examples of multi-link ‘trunks’, not an actual InfiniBand (IB) fabric design.

     

     

    The diagram shows several examples of acceptable configurations. If you scramble links (as shown from B to D) it is more difficult to maintain and it is therefore not recommended.

    Note:

    • When different ‘trunks’ of links connect to the same switch, each trunk must use contiguous ports. Adjacency between different trunks is not required.
    • There is no relationship between the sequence of ports used on one end of a trunk and the sequence used for ports on the other end.
    • Each trunk can have a different number of links, as dictated by the fabric's design.

     

    The next diagram shows examples of how not to cable the fabric.

     

     

    In the illustration each cable trunk between switches has one or more discontinuities, in most cases this condition exists at both ends.  Non-contiguous cabling can cause the routing algorithm to under-utilize the available links by distributing the logical node-to-node paths unequally across the available physical paths. This setup creates unintended blocking. If you introduce multiple discontinuities you can make conditions worse.

     

    Switch to HCA Cabling

    There are no cabling-specific restrictions imposed when cabling HCAs to switches, as long as the inter-switch cabling is correct.  Whether or not HCAs should be connected to a given switch is determined by fabric design rules and is outside the scope of this discussion, see Cabling Considerations for Clos-5 Networks.

     

    Correcting for Sub-Optimal Cabling

    An OpenSM CLI option, port_search_ordering_file, can be used to force the routing algorithm to look at switch ports in a specified order, instead of sequentially.  Non-contiguous physical cabling can be made to appear contiguous (logically) for routing purposes.  The re-ordering is only considered during routing calculations and has no implications otherwise.

    Use the following OpenSM option for port search ordering:

    -O, --port_search_ordering_file <path to file containing port ordering per switch>

     

    The port search ordering file is a text file containing lines using the following syntax:

    <switch GUID including leading ‘0x’> <list of non-zero port numbers separated by spaces>

    The following port_search_ordering_file compensates for the sub-optimal cabling in the previous diagram:

    <GUID of Switch A> 1 2 3 4 5 6 7 9 11 12 13 15 16 17

    <GUID of Switch B> 5 6 7 8 9 10 11 12 26 27 28 29 30 32 33

    <GUID of Switch C> 10 12 14 16 18 20 22 24 11 13 15 17 19 21 23 25

    <GUID of Switch D> 3 4 5 6 7 10 17 18 19 20 21 23 24