7 Replies Latest reply on Jan 31, 2018 4:52 AM by ohammer

    How to create IB partition and test it ?

    ohammer

      Hi,any body did the test?

      my environment :

      server1

      ibstat

      CA 'mlx4_0'

              CA type: MT4099

              Number of ports: 2

              Firmware version: 2.11.500

              Hardware version: 0

              Node GUID: 0xf4521403007dc820

              System image GUID: 0xf4521403007dc823

              Port 1:

                      State: Active

                      Physical state: LinkUp

                      Rate: 56

                      Base lid: 10

                      LMC: 0

                      SM lid: 2

                      Capability mask: 0x02514868

                      Port GUID: 0xf4521403007dc821

                      Link layer: InfiniBand

              Port 2:

                      State: Active

                      Physical state: LinkUp

                      Rate: 56

                      Base lid: 15

                      LMC: 0

                      SM lid: 56

                      Capability mask: 0x02514868

                      Port GUID: 0xf4521403007dc822

                      Link layer: InfiniBand

      --------------------

      server 2

      ibstat

      CA 'mlx4_0'

              CA type: MT4099

              Number of ports: 2

              Firmware version: 2.32.5180

              Hardware version: 0

              Node GUID: 0x248a070300f041f0

              System image GUID: 0x248a070300f041f3

              Port 1:

                      State: Active

                      Physical state: LinkUp

                      Rate: 56

                      Base lid: 25

                      LMC: 0

                      SM lid: 2

                      Capability mask: 0x02514868

                      Port GUID: 0x248a070300f041f1

                      Link layer: InfiniBand

              Port 2:

                      State: Active

                      Physical state: LinkUp

                      Rate: 56

                      Base lid: 4

                      LMC: 0

                      SM lid: 56

                      Capability mask: 0x02514868

                      Port GUID: 0x248a070300f041f2

                      Link layer: InfiniBand

       

      ----------------------------

      ib switch :

      switch-bbb978 [standalone: master] (config) # show ib partition

        Default

          PKey      = 0x7FFF

          defmember = full

          ipoib     = yes

        members

          GUID='ALL' member='default'

       

      under the default partition, ibping is ok .now I do not want to server1 can ping server2 。

       

       

      i confiure like this on switch

       

      switch-bbb978 [standalone: master] (config) # show ib partition

        Default

          PKey      = 0x7FFF

          defmember = full

          ipoib     = yes

        members

          GUID='ALL' member='default'

        test1

          PKey      = 0x0020

          defmember = limited

          ipoib     = no

        members

          GUID='24:8a:07:03:00:f0:41:f1' member='limited'

          GUID='f4:52:14:03:00:7d:c8:21' member='limited'

      if i delete the default partition, other node can be effected.

      if i keep the default partition,server 1 still can ibping server 2 ,how to set the priority ?

        • Re: How to create IB partition and test it ?
          halr

          Default partition is required and all nodes must be part of that partition but they need not be full members. SM/SA node must be full member of default partition. Then none of the nodes will be able to communicate with other nodes except SM/SA node.

           

          You can then create additional partitions supporting IPoIB with groups of nodes as (full) members which can ping each other.

            • Re: How to create IB partition and test it ?
              ohammer

              thks.

              i configure my switch like this ,

               

              first  remove default partition -- no ib partition Default

              then create partition like below

              switch-bbb978 [standalone: master] (config) # show ib partition

                test1

                  PKey      = 0x0005

                  defmember = full

                  ipoib     = yes

                members

                  GUID='24:8a:07:03:00:dd:c9:01' member='limited'

                  GUID='24:8a:07:03:00:f0:41:f1' member='full'

                test9

                  PKey      = 0x0009

                  defmember = full

                  ipoib     = yes

                members

                  GUID='24:8a:07:03:00:dd:ba:21' member='default'

                  GUID='f4:52:14:03:00:82:81:51' member='default'

               

               

              the server in each partition can ping each other with the child interface ib0.8005,but can not ping each other with ib0,and cannot ibping -L lid .

               

              Is this configuration right?

                • Re: How to create IB partition and test it ?
                  halr

                  The fact that there is no default partition probably explains why ibping fails.

                   

                  The default partition must be present. The way to isolate the nodes is to make them all limited members of this partition with the exception of the SM node which must be full member of that partition (0x7fff). Even in this configuration, ibping will fail as limited member cannot talk to another limited member.

                   

                  Also, I don't understand why limited member is being configured on partition 0x0009.

                   

                  xu chen wrote:

                   

                  thks.

                  i configure my switch like this ,

                   

                  first remove default partition -- no ib partition Default

                  then create partition like below

                  switch-bbb978 [standalone: master] (config) # show ib partition

                  test1

                  PKey = 0x0005

                  defmember = full

                  ipoib = yes

                  members

                  GUID='24:8a:07:03:00:dd:c9:01' member='limited'

                  GUID='24:8a:07:03:00:f0:41:f1' member='full'

                  test9

                  PKey = 0x0009

                  defmember = full

                  ipoib = yes

                  members

                  GUID='24:8a:07:03:00:dd:ba:21' member='default'

                  GUID='f4:52:14:03:00:82:81:51' member='default'

                   

                   

                  the server in each partition can ping each other with the child interface ib0.8005,but can not ping each other with ib0,and cannot ibping -L lid .

                   

                  Is this configuration right?

              • Re: How to create IB partition and test it ?
                alkx

                Hi, partitions are IPoIB interfaces and do not limit IB communications between the nodes. You might try the following,

                Unload ib_ipoib module ( you'll have no ibX interfaces)

                ibping between the nodes

                 

                Note: for some reason, you have two different subnet managers on the nodes (see sm lid in ibstat output). I would suggest explicitly specify interface/port when running ibping/ibserver or disconnect the other second port on each node.