6 Replies Latest reply on Apr 13, 2016 4:27 AM by jsm820

    I have a trouble that I can't change Ethernet to Infiniband.

    jsm820

      I am trying to connect two machines with infiniband (MCX353A-QCBT) in Ubuntu 14.04. But I failed.

       

      This is my state. (lspci, ifconfig, ibstat, ibnodes, /sbin/connectx_port_config, dmesg). Does anyone have any directions, links, or advice?

       

      sangmuk@sclab1:~$ lspci | grep Mellanox

      83:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

       

      sangmuk@sclab1:~$ ifconfig

      eth0      Link encap:Ethernet  ..

      eth1      Link encap:Ethernet  ..

      eth2      Link encap:Ethernet  HWaddr 7c:fe:90:16:8b:50

                inet6 addr: fe80::7efe:90ff:fe16:8b50/64 Scope:Link

                UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

                RX packets:126402 errors:0 dropped:0 overruns:0 frame:0

                TX packets:126585 errors:0 dropped:0 overruns:0 carrier:0

                collisions:0 txqueuelen:1000

                RX bytes:23725053 (23.7 MB)  TX bytes:23771526 (23.7 MB)

      lo        Link encap:Local Loopback

                inet addr:127.0.0.1  Mask:255.0.0.0

                inet6 addr: ::1/128 Scope:Host

                UP LOOPBACK RUNNING  MTU:65536  Metric:1

                RX packets:390193 errors:0 dropped:0 overruns:0 frame:0

                TX packets:390193 errors:0 dropped:0 overruns:0 carrier:0

                collisions:0 txqueuelen:0

                RX bytes:26791107 (26.7 MB)  TX bytes:26791107 (26.7 MB)

       

      sangmuk@sclab1:~$ ibstat

      CA 'mlx4_0'

              CA type: MT4099

              Number of ports: 1

              Firmware version: 2.36.5000

              Hardware version: 1

              Node GUID: 0x7cfe900300168b50

              System image GUID: 0x7cfe900300168b53

              Port 1:

                      State: Active

                      Physical state: LinkUp

                      Rate: 10

                      Base lid: 0

                      LMC: 0

                      SM lid: 0

                      Capability mask: 0x0c010000

                      Port GUID: 0x7efe90fffe168b50

                      Link layer: Ethernet

       

      sangmuk@sclab1:~$ ibnodes

      ibwarn: [27593] mad_rpc_open_port: client_register for mgmt 1 failed

      src/ibnetdisc.c:766; can't open MAD port ((null):0)

      /usr/sbin/ibnetdiscover: iberror: failed: discover failed

      ibwarn: [27598] mad_rpc_open_port: client_register for mgmt 1 failed

      src/ibnetdisc.c:766; can't open MAD port ((null):0)

      /usr/sbin/ibnetdiscover: iberror: failed: discover failed

       

      sangmuk@sclab1:~$ ibhosts

      ibwarn: [27635] mad_rpc_open_port: client_register for mgmt 1 failed

      src/ibnetdisc.c:766; can't open MAD port ((null):0)

      /usr/sbin/ibnetdiscover: iberror: failed: discover failed

       

      sangmuk@sclab1:~$ sudo /sbin/connectx_port_config

      ConnectX PCI devices :

      |----------------------------|

      | 1             0000:83:00.0 |

      |----------------------------|

      Before port change:

      eth

      |----------------------------|

      | Possible port modes:       |

      | 1: Infiniband              |

      | 2: Ethernet                |

      | 3: AutoSense               |

      |----------------------------|

      Select mode for port 1 (1,2,3): 1

      WARNING: Illegal port configuration attempted,

              Please view dmesg for details.

       

      sangmuk@sclab1:~$ dmesg

      ...

      [338561.776021] mlx4_core 0000:83:00.0: Requested port type for port 1 is not supported on this HCA

        • Re: I have a trouble that I can't change Ethernet to Infiniband.
          eddie.notz

          Hi Sangmuk Jo,

           

          Can you please send the output of:

          "ibv_devinfo"

          or

          1. mst start

          2. flint -d mlx4_0 q

            • Re: I have a trouble that I can't change Ethernet to Infiniband.
              jsm820

              Hi!

               

              My "ibv_devinfo" and "flint".

               

              sangmuk@sclab1:~$ ibv_devinfo

              hca_id: mlx4_0

                      transport:                      InfiniBand (0)

                      fw_ver:                         2.36.5000

                      node_guid:                      7cfe:9003:0016:8b50

                      sys_image_guid:                 7cfe:9003:0016:8b53

                      vendor_id:                      0x02c9

                      vendor_part_id:                 4099

                      hw_ver:                         0x1

                      board_id:                       MT_1060110018

                      phys_port_cnt:                  1

                      Device ports:

                              port:   1

                                      state:                  PORT_ACTIVE (4)

                                      max_mtu:                4096 (5)

                                      active_mtu:             1024 (3)

                                      sm_lid:                 0

                                      port_lid:               0

                                      port_lmc:               0x00

                                      link_layer:             Ethernet

               

              sangmuk@sclab1:~$ flint -d mlx4_0 q

              Image type:          FS2

              FW Version:          2.36.5000

              FW Release Date:     26.1.2016

              Product Version:     02.36.50.00

              Rom Info:            type=PXE version=3.4.718 devid=4099

              Device ID:           4099

              Description:         Node             Port1            Port2            Sys image

              GUIDs:               7cfe900300168b50 7cfe900300168b51 7cfe900300168b52 7cfe900300168b53

              MACs:                                     7cfe90168b50     7cfe90168b51

              VSD:

              PSID:                MT_1060110018