6 Replies Latest reply on Sep 16, 2014 11:17 PM by ophirmaor

    Troubleshooting the iSER enable on Mellanox ConnectX-2 EN and running iSCSI traffic.

    am2014

      Hi,

       

      I have folowing card on my system.

      lspci | grep -i mellanox

      02:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)

       

      I am running CentOs 6.5.

      uname -a

      Linux RV-189 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

      cat /etc/issue

      CentOS release 6.5 (Final)

      Kernel \r on an \m

       

      I am following the link .

      HowTo setup RoCE connection using Inbox driver (RHEL, Ubuntu)

       

      I do not see the enp5s0 or any other interface which can describe the Mellanox card on my system.

      #ifconfig enp5s0 12.12.12.1/24 up

       

      Can someone pl let me know what am I missing?

       

      Thanks in advance,

      Anil

        • Re: Troubleshooting the iSER enable on Mellanox ConnectX-2 EN and running iSCSI traffic.
          ophirmaor

          Hi Anil,

          Do you have MLNX_OFED installed? did you use OFED inbox driver?

          Note that the example is from RHEL7 and over there the interface name is different (not eth1,eth2 ..) but enp5s0 or similar.

          try

           

          #yum -y groupinstall "InfiniBand Support"

          #yum -y install perftest infiniband-diags

          - for the inbox driver, it may be the easiest.

           

          Please send me the output of

          #ibdev2netdev

          #ofed_info | head

          you didn't reach the level of testing ISER, first you need to make sure that RoCE is running, and for that you need RDMA support.


          Ophir.

            • Re: Troubleshooting the iSER enable on Mellanox ConnectX-2 EN and running iSCSI traffic.
              am2014

              Thanks Ophir for the reply.

               

              Hi Anil,

               

              Do you have MLNX_OFED installed? did you use OFED inbox driver?

               

              Anil> Whether MLNX_OFED should be available as inbox? I tried your commands

              and they did not run, please see the output given below.

               

              # yum install "InfiniBand Group"

              Loaded plugins: aliases, changelog, downloadonly, fastestmirror, kabi,

              presto, refresh-packagekit,

                            : security, tmprepo, verify, versionlock

              Loading support for CentOS kernel ABI

              Loading mirror speeds from cached hostfile

               

              • base: mirrors.unifiedlayer.com

               

              • extras: mirror.anl.gov

               

              • updates: mirrors.easynews.com

              Setting up Install Process

              No package InfiniBand Group available.

              Error: Nothing to do

              #

              #

              # ibdev2netdev

              bash: ibdev2netdev: command not found

              # ibdev2netdev

              bash: ibdev2netdev: command not found

              # ofed_info | head

              bash: ofed_info: command not found

              #

               

               

               

              you didn't reach the level of testing ISER, first you need to make sure

              that RoCE is running, and for that you need RDMA support.

               

              Anil>> I knew, I am missing something. yes, I want to make sure that RoCE

              is running first

               

              Thanks,

               

              Anil

                • Re: Troubleshooting the iSER enable on Mellanox ConnectX-2 EN and running iSCSI traffic.
                  ophirmaor

                  Run

                   

                  # yum install "InfiniBand Support"

                   

                  See this post

                   

                  HowTo Setup RDMA Connection using Inbox Driver (RHEL, Ubuntu)

                   

                  You can also install MLNX_OFED, see Mellanox website for the driver package.

                  Click here, and follow the links.

                  Mellanox Products: Mellanox OpenFabrics Enterprise Distribution for Linux (MLNX_OFED)

                   

                   

                  Thanks,

                  Ophir.

                    • Re: Troubleshooting the iSER enable on Mellanox ConnectX-2 EN and running iSCSI traffic.
                      am2014

                      Ophir,

                       

                      This <http://community.mellanox.com/docs/DOC-1465> is the exact link what I

                      was following. I installed packages from there and got stuck up at

                      following section.

                       

                      Configure port parameters:

                      Configure IP Address and enable the port.

                      *It can be done via console scripts such as nmtui, ifconfig (not permanent)

                      or other method*

                      For example:

                       

                      #ifconfig enp5s0 12.12.12.1/24 <http://12.12.12.1/24> up

                       

                       

                       

                      I tried to install these packages again and it came up with following

                      response:

                       

                      # yum -y groupinstall "InfiniBand Support"

                      Loaded plugins: aliases, changelog, downloadonly, fastestmirror, kabi,

                      presto, refresh-packagekit, security, tmprepo, verify,

                                    : versionlock

                      Loading support for CentOS kernel ABI

                      Loading mirror speeds from cached hostfile

                       

                      • base: centos-distro.cavecreek.net

                       

                      • extras: mirror.nwresd.org

                       

                      • updates: mirrors.xmission.com

                      Setting up Group Process

                      Checking for new repos for mirrors

                      Package libibverbs-utils-1.1.7-1.el6.x86_64 already installed and latest

                      version

                      Package libibverbs-1.1.7-1.el6.x86_64 already installed and latest version

                      Package librdmacm-utils-1.0.17-1.el6.x86_64 already installed and latest

                      version

                      Package libibcm-1.0.5-3.el6.x86_64 already installed and latest version

                      Package rdma-3.10-3.el6.noarch already installed and latest version

                      Package librdmacm-1.0.17-1.el6.x86_64 already installed and latest version

                      Package libmlx4-1.0.5-4.el6.1.x86_64 already installed and latest version

                      Package libipathverbs-1.2-4.el6.x86_64 already installed and latest version

                      Package libmthca-1.0.6-3.el6.x86_64 already installed and latest version

                      Package libnes-1.1.3-1.el6.x86_64 already installed and latest version

                      Package libcxgb3-1.3.1-1.el6.x86_64 already installed and latest version

                      Package ibsim-0.5-7.el6.x86_64 already installed and latest version

                      Package dapl-2.0.34-1.el6.x86_64 already installed and latest version

                      Package libibmad-1.3.9-1.el6.x86_64 already installed and latest version

                      Package libibumad-1.3.8-1.el6.x86_64 already installed and latest version

                      Package rds-tools-2.0.6-3.el6.x86_64 already installed and latest version

                      Package ibutils-1.5.7-8.el6.x86_64 already installed and latest version

                      Package ibacm-1.0.8-0.git7a3adb7.el6.x86_64 already installed and latest

                      version

                      Warning: Group infiniband does not have any packages.

                      No packages in any requested group available to install or update

                      #

                      #

                      # yum -y install perftest infiniband-diags

                      Loaded plugins: aliases, changelog, downloadonly, fastestmirror, kabi,

                      presto, refresh-packagekit, security, tmprepo, verify,

                                    : versionlock

                      Loading support for CentOS kernel ABI

                      Loading mirror speeds from cached hostfile

                       

                      • base: mirror.tocici.com

                       

                      • extras: mirror.nwresd.org

                       

                      • updates: mirrors.xmission.com

                      Setting up Install Process

                      Package perftest-2.0-2.el6.x86_64 already installed and latest version

                      Package infiniband-diags-1.5.12-5.el6.x86_64 already installed and latest

                      version

                      Nothing to do

                      #

                       

                       

                      So it seems like theses packages are there on my system.

                       

                      If you think, I need to do following extra, I can do that.

                       

                      *You can also install MLNX_OFED, see Mellanox website for the driver

                      package.*

                       

                      Click here, and follow the links.

                       

                      *Mellanox Products: Mellanox OpenFabrics Enterprise Distribution for Linux

                      (MLNX_OFED)*

                      <http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers>

                       

                       

                       

                       

                      On Tue, Sep 16, 2014 at 12:02 PM, ophirmaor <socialmedia@mellanox.com>

                  • Re: Troubleshooting the iSER enable on Mellanox ConnectX-2 EN and running iSCSI traffic.
                    am2014

                    Ophir,

                     

                    I downloaded driver from your web site and probably it seems to behave much

                    better now. I am giving you output of the commands what you asked earlier.

                     

                    # ibdev2netdev

                    mlx4_0 port 2 ==> eth4 (Down)

                    mlx4_0 port 1 ==> eth5 (Down)

                    # ofed_info | head

                    MLNX_OFED_LINUX-2.3-1.0.1 (OFED-2.3-1.0.0):

                     

                    ar_mgr:

                    ar_mgr/e338fbc05dd6b480f8505791e50700b7/ar_mgr-1.0-0.23.g20d8d45.tar.gz

                     

                    bupc:

                    bupc/6b0f0c8b7d7ff2e258e600d65f6da291/bupc-2.18.0-421.src.rpm

                     

                    cc_mgr:

                    cc_mgr/4a29eb085479e51b6bb278efcddfe379/cc_mgr-1.0-0.22.g20d8d45.tar.gz

                    #

                     

                     

                     

                    So interface eth4 and eth5 can be used for iSER, correct?

                     

                    I have two cards and want to set one card in one system as iSER Initiator

                    and other card as iSER target.

                     

                    Can I use these two cards in the same system as Initiator and target?

                     

                    Thanks,

                    Anil

                  • Re: Troubleshooting the iSER enable on Mellanox ConnectX-2 EN and running iSCSI traffic.
                    ophirmaor

                    eth4 and eth5 will be the interfaces you need to configure. correct.

                    Do you have two hosts? if so, you can configure one host (with one Mellanox adapter) as target (you can use TGT for example) and another server equipped with Mellanox adapter as the initiator.

                    you can follow HowTo Configure TGT enabled with iSER Transport for RHEL for that.

                     

                    Ophir.