0 Replies Latest reply on May 11, 2018 1:39 AM by aob

    Ubuntu 16.04 InfiniHost III Driver

    aob

      Hi all

       

      Thanks in advance for your feedback.

      Before starting this discussion, I've done the following:

      # Read relevant manuals

      # Searched stackexchanges, online forums and communities (including this one).

      The underlying OS is Ubuntu 16.04. I'm fully aware that CentOS 7 with the "Infiniband Support" group installation might solve the issue. However, changing the OS is not a possibility.

       

      In what follows, I've listed some relevant information:

      $ lspci | grep Mellanox

      02:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 20)

      $ /etc/init.d/openibd restart

      Unloading HCA driver:                                               [  OK  ]

      Loading HCA driver and Access Layer:                       [  OK  ]

      Check the attachment for Openibd configuration file.

      Driver installation: (MLNX_OFED_LINUX-4.3-1.0.1.0-ubuntu16.04-x86_64)

      $ ./mlnxofedinstall --without-dkms --without-fw-update --add-kernel-support --kernel 4.4.0-124-generic

       

      $ lsmod | grep -i ib

      ib_ucm                 20480  0

      ib_ipoib              159744  0

      ib_cm                  53248  3 rdma_cm,ib_ucm,ib_ipoib

      ib_uverbs             102400  2 ib_ucm,rdma_ucm

      ib_umad                24576  0

      mlx5_ib               249856  0

      mlx5_core             753664  2 mlx5_ib,mlx5_fpga_tools

      mlx4_ib               208896  0

      ib_core               262144  10 rdma_cm,ib_cm,iw_cm,mlx4_ib,mlx5_ib,ib_ucm,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib

      mlx4_core             344064  2 mlx4_en,mlx4_ib

      mlx_compat             24576  15 rdma_cm,ib_cm,iw_cm,mlx4_en,mlx4_ib,mlx5_ib,mlx5_fpga_tools,ib_ucm,ib_core,ib_umad,ib_uverbs,mlx4_core,mlx5_core,rdma_ucm,ib_ipoib

      libiscsi_tcp           24576  1 iscsi_tcp

      libiscsi               53248  2 libiscsi_tcp,iscsi_tcp

      scsi_transport_iscsi   102400  3 iscsi_tcp,libiscsi

      libcrc32c              16384  1 raid456

      I've also tried manually loading ib_mthca as well as including it in the openibd.conf, but the latter resulted in a warning message

      please remove MTHCA_LOAD from your /etc/infiniband/openib.conf file

      $ /etc/init.d/openibd status

       

       

       

        HCA driver loaded

       

       

       

       

      The following OFED modules are loaded:

       

       

        rdma_ucm

        rdma_cm

        ib_ipoib

        mlx4_core

        mlx4_ib

        mlx4_en

        mlx5_core

        mlx5_ib

        ib_uverbs

        ib_umad

        ib_ucm

        ib_cm

        ib_core

        mlxfw

        mlx5_fpga_tools

      Last but not least, only for completeness sake:

      $ dmesg | grep -i ib

      [    0.000000] tsc: Fast TSC calibration using PIT

      [    0.000068] Calibrating delay loop (skipped), value calculated using timer frequency.. 4623.83 BogoMIPS (lpj=9247668)

      [    0.033636] FEATURE IBPB Not Present

      [    0.036146] Spectre V2 mitigation: Speculation control IBPB not-supported IBRS not-supported

      [    0.402903] vgaarb: bridge control possible 0000:01:01.0

      [    0.403416] libata version 3.00 loaded.

      [    1.249949] IBS: LVT offset 1 assigned

      [    1.250066] perf: AMD IBS detected (0x00000007)

      [    1.306910] libphy: Fixed MDIO Bus: probed

      [    1.397806] PM: Hibernation image not present or could not be loaded.

      [    2.004411] sd 2:0:0:0: [sda] 488390625 512-byte logical blocks: (250 GB/233 GiB)

      [    2.247252] tsc: Refined TSC clocksource calibration: 2311.847 MHz

      [   10.552786] systemd[1]: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP + IBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)

      [   12.447069] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe.

      [   22.681305] audit: type=1400 audit(1525876878.782:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=878 comm="apparmor_parser"

      [   22.681319] audit: type=1400 audit(1525876878.782:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=878 comm="apparmor_parser"

      [   23.004756] audit: type=1400 audit(1525876879.102:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/lxd/lxd-bridge-proxy" pid=877 comm="apparmor_parser"