4 Replies Latest reply on Oct 11, 2018 3:07 PM by nmotihar

    Support for "INBOX drivers?" for 18.04/connected mode?

    billbroadley

      I saw at http://www.mellanox.com/page/products_dyn?product_family=26

      "Linux Inbox Drivers

      Mellanox Adapters' Linux VPI Drivers for Ethernet and InfiniBand are also available Inbox in all the major distributions, RHEL, SLES, Ubuntu and more. Inbox drivers enable Mellanox High performance for Cloud, HPC, Storage, Financial Services and more with the Out of box experience of Enterprise grade Linux distributions."

      I've used this for several generations of Mellanox cards of for last decade with a wide variety of linux distributions.  Just config with /etc/network/interfaces, ifconfig ib0 works, datagram/connected mode works, IPoIB works. 

      From what I can tell with 18.04 it doesn't work.  The default (ubuntu supplied/INBOX) drivers:

      # cat /sys/class/net/ib0/mode

      datagram

      # echo connected > /sys/class/net/ib0/mode

      -bash: echo: write error: Invalid argument


      I found docs that with the MLNX_OFED drivers that you just:

        # cat ib_ipoib.conf

        options ib_ipoib ipoib_enhanced=0

      I get:

      [   57.573664] ib_ipoib: unknown parameter 'ipoib_enhanced' ignored


      Is there any documentation for the "INBOX drivers"?  Anyone know how to get connected mode working?  I'm guess the 18.04 drivers are too old to have the ability to disable enhanced mode, but to new to have the connected mode working by default.

        • Re: Support for "INBOX drivers?" for 18.04/connected mode?
          nmotihar

          Hi Bill,

           

          Here is the link to the Release Notes for Ubuntu 18.04 Inbox Driver. Section 2 Changes and New Features lists the support for Enhanced IPoIB for ConnectX-4 cards. Also, I am attaching the link for the User Manual for your reference. Which card are you using? Also, you mentioned you are receiving the following error [   57.573664] ib_ipoib: unknown parameter 'ipoib_enhanced' ignored. Is this after testing with MLNX_OFED? If yes, which OFED version did you test it with? Also, for any questions related Inbox Driver, it would be great to reach out to OS Vendor.

           

          http://www.mellanox.com/pdf/prod_software/Ubuntu_18.04_Inbox_Driver_Release_Notes.pdf

           

          http://www.mellanox.com/pdf/prod_software/Ubuntu_18_04_Inbox_Driver_User_Manual.pdf

            • Re: Support for "INBOX drivers?" for 18.04/connected mode?
              billbroadley

              Hi Bill,

               

               

               

              Here is the link to the Release Notes for Ubuntu 18.04 Inbox Driver. Section 2

              Changes and New Features lists the support for Enhanced IPoIB for ConnectX-4

              cards. Also, I am attaching the link for the User Manual for your reference.

               

              The documentation was quite helpful, thanks.

               

              Which card are you using?

               

              root@k1:~# lspci | grep -i mellanox

              06:00.0 Infiniband controller: Mellanox Technologies MT27700 Family

               

               

              Also, you mentioned you are receiving the following

              error  ib_ipoib: unknown parameter 'ipoib_enhanced' ignored. Is

              this after testing with MLNX_OFED? If yes, which OFED version did you test it with?

               

              I was tried what comes with ubuntu 18.04 and with

              "mlnx-en-4.4-2.0.7.0-ubuntu18.04-x86_64".  I'm open to a solution with either.

               

              Do you have any recommendations for getting ipoib connected mode to work with

              either the native ubuntu 18.04 drivers or the MLNX_OFED drivers?

               

              I tried this because it was recommended in the Mellanox forum:

                root@k2:/etc/modprobe.d# cat ib_ipoib.conf

                options ib_ipoib ipoib_enhanced=0

               

              But when I boot the above mentioned error shows up.

               

              When I install the MLNX_OFED driver, I don't see any errors.  Here's the summary:

              Device #1:

                • Re: Support for "INBOX drivers?" for 18.04/connected mode?
                  billbroadley

                  I had the wrong package installed, MLNX_OFED fixed the problem and connected mode now works.

                   

                  Is there any way to get connected mode working with the drivers that come with Ubuntu 18.04?

                    • Re: Support for "INBOX drivers?" for 18.04/connected mode?
                      nmotihar

                      Hi Bill,

                       

                      It would be great if you could run the following commands to validate if the parameter ipoib_enhanced is supported on your system with inbox driver:

                       

                      1. #find /lib/modules -iname '*ib_ipoib*'

                      (Example output from host running Ubuntu 16.04)

                      /lib/modules/4.4.0-134-generic/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko

                      /lib/modules/4.4.0-116-generic/updates/dkms/ib_ipoib.ko

                       

                      The first line denotes Inbox driver and second line denotes mlnx driver

                       

                      2.#modinfo  /lib/modules/4.4.0-134-generic/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko

                       

                      Check which parameters are supported(You will see in output, lines that start with "parm:").

                       

                      Compare the same by running modinfo with one which has mlnx driver

                       

                      #modinfo  /lib/modules/4.4.0-116-generic/updates/dkms/ib_ipoib.ko

                       

                      Please share the outputs.