0 Replies Latest reply on Apr 23, 2018 10:52 AM by db808

    Enabling second Infiniband adapter for IPoIB under CentOS 7.4

    db808

      Running CentOS 7.4 with MLNX_OFED_LINUX-4.3-1.0.1.0 (OFED-4.3-1.0.1)

       

      with TWO dual-port ConnectX-4 100Gbit EDR adapters for 4 ports total.

       

      ibstats shows all 4 ports

       

      ibstat -l

      mlx5_0

      mlx5_1

      mlx5_2

      mlx5_3

       

      ibdev2netdev -v

      0000:17:00.0 mlx5_0 (MT4115 - MT1609X08073) CX456A - ConnectX-4 QSFP28 fw 12.22.1002 port 1 (ACTIVE) ==> ib0 (Up)

      0000:17:00.1 mlx5_1 (MT4115 - MT1609X08073) CX456A - ConnectX-4 QSFP28 fw 12.22.1002 port 1 (ACTIVE) ==> ib1 (Up)

      0000:65:00.0 mlx5_2 (MT4115 - MT1545X04735) CX456A - ConnectX-4 QSFP fw 12.22.1002 port 1 (ACTIVE) ==> ib2 (Down)

      0000:65:00.1 mlx5_3 (MT4115 - MT1545X04735) CX456A - ConnectX-4 QSFP fw 12.22.1002 port 1 (ACTIVE) ==> ib3 (Down)

       

      All 4 ports are active for the SRP protocol for storage.

       

      There are scripts in /etc/sysconfig/network-scripts for "ifcfg-ib0" and "ifcfg-ib1" but not for devices ib2 and ib3.

       

      How do you get the initial "ifcfg" scripts created for ib2 and ib3?  The critical setting in these files is the UUID for the IPoIB device, which I don't know how to query or generate.

       

      I suspect there is a config file setting somewhere that needs to be set, but I can't determine which configuration setting is limiting the IBoIP scan to only 2 devices.

       

      There ARE entries in /sys/class/net for all for devices ....ib0, ib1, ib2, and ib3:

      ls -al /sys/class/net

      total 0

      drwxr-xr-x  2 root root 0 Apr 23 09:24 .

      drwxr-xr-x 63 root root 0 Apr 23 08:28 ..

      lrwxrwxrwx  1 root root 0 Apr 23 09:24 em1 -> ../../devices/pci0000:00/0000:00:1f.6/net/em1

      lrwxrwxrwx  1 root root 0 Apr 23 09:24 ib0 -> ../../devices/pci0000:16/0000:16:00.0/0000:17:00.0/net/ib0

      lrwxrwxrwx  1 root root 0 Apr 23 09:24 ib1 -> ../../devices/pci0000:16/0000:16:00.0/0000:17:00.1/net/ib1

      lrwxrwxrwx  1 root root 0 Apr 23 09:24 ib2 -> ../../devices/pci0000:64/0000:64:00.0/0000:65:00.0/net/ib2

      lrwxrwxrwx  1 root root 0 Apr 23 09:24 ib3 -> ../../devices/pci0000:64/0000:64:00.0/0000:65:00.1/net/ib3

      lrwxrwxrwx  1 root root 0 Apr 23 09:24 lo -> ../../devices/virtual/net/lo

      lrwxrwxrwx  1 root root 0 Apr 23 09:24 p1p1 -> ../../devices/pci0000:b2/0000:b2:00.0/0000:b3:00.0/net/p1p1

      lrwxrwxrwx  1 root root 0 Apr 23 09:24 p1p2 -> ../../devices/pci0000:b2/0000:b2:00.0/0000:b3:00.1/net/p1p2

       

      Thanks for your help.

       

      ****************************

      Follow-on ... I found that the command "uuidgen" can be used to generate the UUIDs for ib2 and ib3 .... and with this information I can create the ifcfg-ib2 and ifcfg-ib3 files

       

      This should solve my problem, but does not identify why the files were not created in the first place.