4 Replies Latest reply on May 22, 2017 12:03 PM by alpha-linux

    MCX354A-FCBT / ConnectX-3 in Ethernet mode

    alpha-linux

      Hi all

       

      I'm pretty new to Mellanox adapters. Just got some ConnectX-3 VPI (MCX354A-FCBT) adapters and want to use the in my VMware homelab in 40G Ethernet mode. (no IB)

      my plan is to connect the ESXi host to a Nexus switch with 40G QSFP ports.

      is there some tutorial or best practice guide to accomplish this ?

      - which firmware to use

      - what drivers to load in ESXi 6

      - how to change ConnectX-3 ports to 40G Ethernet mode

      - what QSFP-cables to use (prefer Copper) / will this one work with a Nexus-switch ? (MC2207128-003 - FDR InfiniBand QSFP passive copper cable, 3m)

       

      feedback is very much appreciated

      regards,

        • Re: MCX354A-FCBT / ConnectX-3 in Ethernet mode
          inbusiness

          Hi!


          At first you must update latest MCX354A-FCBT firmware then change to ETH mode with firmware tools.


          Then you can use your hca like general ethernet adapter on any Operating Systems that supported from Mellanox driver.

          • Re: MCX354A-FCBT / ConnectX-3 in Ethernet mode
            karen

            Hi Mark,

            For Connectx-3 and Esxi 6.0 please install the MLNX-OFED-ESX  3.15.5.5 version, for your reference please see the following download link:

            http://www.mellanox.com/page/products_dyn?product_family=29&mtag=vmware_driver

            Please note that you will also need to upgrade the card firmware version to 2.36.5000 and above.

            For this you would need to use our MFT tool package to query and burn new firmware if needed, this can be downloaded from here:

            http://www.mellanox.com/page/management_tools?mtag=firmware_tools

             

            For changing the speed on the host side please see the following article from VMware:

            https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004089

            if the connection is through a switch please also confirm that the switch port is configured to 40 G speed.

             

            Thanks,

            Karen.

             

              • Re: MCX354A-FCBT / ConnectX-3 in Ethernet mode
                alpha-linux

                Hi karen


                thanks for feedback !
                just installed MFT and MST (mft-4.6.0.48-10EM-600.0.0.2494585.x86_64.vib and net-mst_4.6.0.48-1OEM.600.0.0.2494585.vib)

                Installation Result
                  Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
                  Reboot Required: true
                  VIBs Installed: Mellanox_bootbank_mft_4.6.0.48-0

                Installation Result

                   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

                   Reboot Required: true

                   VIBs Installed: MEL_bootbank_net-mst_4.6.0.48-1OEM.600.0.0.2494585

                 

                but got some issue when using mst; it says mst is loaded but when issuing "mst status" it says it is not running ?

                adapter is present and running, see below:

                 

                start mst by

                [esxi-02:~] /opt/mellanox/bin/mst start
                Module mst loaded successfully

                [esxi-02:~] /opt/mellanox/bin/mst status -v
                    No MST devices were found or MST modules are not loaded.

                    You may need to run 'mst start' to load MST modules.

                [esxi-02:~] /opt/mellanox/bin/mst start

                Module mst is already loaded

                 

                [esxi-02:~] lspci | grep Mellanox

                0000:01:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] [vmnic2]

                 

                [esxi-02:~] esxcli software vib list | grep Mellanox

                mft     4.6.0.48-0     Mellanox  PartnerSupported  2017-05-21

                 

                [esxi-02:~] esxcli system module list | grep mlx4

                nmlx4_core     true        true

                nmlx4_en        true        true

                nmlx4_rdma    true        true

              • Re: MCX354A-FCBT / ConnectX-3 in Ethernet mode
                alpha-linux

                works now; choose the wrong type. with type "natiive" instead "vmk" it does the trick.

                mft-4.6.0.48-10EM-600.0.0.2768847.x86_64.vib
                nmst-4.6.0.48-1OEM.600.0.0.2768847.x86_64.vib

                 

                running now with latest fw:

                /opt/mellanox/bin/flint -d mt4099_pci_cr0 query

                Image type:          FS2

                FW Version:          2.40.7000

                FW Release Date:     22.3.2017

                Product Version:     02.40.70.00

                Rom Info:            type=PXE version=3.4.746 devid=4099

                Device ID:           4099

                 

                Also set "LINK_TYPE" to Ethernet by

                /opt/mellanox/bin/mlxconfig -d mt4099_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2