Understanding the Root GUID File (SM Configuration)

Version 5

    This post describes how to use the root_guid_file parameter as a part of the SM configuration.

    This post is basic and meant for people that have basic understanding in InfiniBand.

     

    References

     

    Overview

    The root_guid_file parameter is a path to the file that contains the root (top-level) spine switches used by the opensm fat tree or up/down routing algorithms (only those). The routing algorithm will set the root nodes to prevent credit loops or a loop-deadlock in the fabric.

    In most cases the root GUIDs correspond to the Node GUIDs of spine switches in the fabric topology.

    If this parameter is not configured:

    - With updn routing engine, the SM will attempt to detect the root in the fabric. This can lead to less efficient unicast routing calculated by the opensm.

    - With ftree routing engine, the routing calculation might fail to the Minhop routing engine if the underlying topology is not a purely theoretical Fat-Tree.

     

    It is highly recommended that you use the root_guid_file option as it contains the root nodes for ranking.

     

    Note: If some of the root switches were replaced or more root switches were added to the fabric, the root_guid_file should be updated accordingly to maintain efficient routing in the fabric.

    After the file is updated, you should run “pkill –HUP opensm” to apply the changes without restarting the OpenSM application.

     

    Use one of the three options to add the root GUIDs to the file. Deciding which one to use depends on the way you are running the SM:

    • From MLNX OS
    • From the Host
    • From the UFM server

     

    Configuration Options

     

    From the Host

    1. Edit the opensm.conf file as follows:

    routing_engine updn

    root_guid_file /etc/opensm/root_guid.conf

     

    2. Create the root_guid. conf file in that location and add the relevant GUIDs to it:

    # cat  /etc/opensm/root_guid.conf

    0x7cfe900300172a52

    0x7cfe900300172a53

    0x7cfe900300172a54

     

    3. Restart the driver:

    # /etc/init.d/opensmd stop

    # /etc/init.d/opensmd start

     

    From the Switch

    1. Set the routing engine to up/down:

    switch  (config) # ib sm routing-engines ftree updn minhop

    switch   (config) # show ib sm routing-engines

    ftree, updn, minhop

     

    2. Add root GUIDs as shown in the example:

    switch  (config)# ib sm root-guid 7c:fe:90:03:00:17:2a:52

    switch  (config)# ib sm root-guid 7c:fe:90:03:00:16:94:e3

    switch  (config)# ib sm root-guid 7c:fe:90:03:00:17:30:c4

    switch  (config) # show ib sm root-guid

    7c:fe:90:03:00:17:2a:52

    7c:fe:90:03:00:16:94:e3

    7c:fe:90:03:00:17:30:c4

     

    From the Unified Fabric Manager (UFM)

    1. Edit the opensm.conf file as follows:

    routing_engine updn

    root_guid_file /opt/ufm/files/conf/opensm/root_guid.conf

     

    2. Create the root_guid.conf file and add the GUIDs to it:

    # cat /opt/ufm/files/conf/opensm/root_guid.conf

    0x7cfe900300172a52

    0x7cfe900300172a53

    0x7cfe900300172a54

     

    3. Restart the driver:

    /etc/init.d/ufmd stop

    /etc/init.d/ufmd start