5 Replies Latest reply on Jul 8, 2014 6:26 AM by eddie.notz

    56 GbE over a ConnectX-3 VPI adapter

      Hello,

       

      I have a pair of ConnectX-3 VPI adapters (MCX354A-FBCT) in Ethernet mode and am attempting to test out 56 GbE connectivity by directly connecting the adapters to one another via an FDR cable (MC2207128). However, the cards only establish link at 40 GbE. In fact, ethtool at least does not seem to report that the card is 56 GbE-capable. I am using firmware 2.31.5050 and driver 2.2-1.0.0. For example, ethtool shows the following:

       

      Supported link modes:   1000baseT/Full
                              1000baseKX/Full
                              10000baseT/Full
                              10000baseKX4/Full
                              10000baseKR/Full
                              40000baseKR4/Full
                              40000baseCR4/Full
                              40000baseSR4/Full
                              40000baseLR4/Full

       

      Looking at the some of the ethtool source (drivers/net/ethernet/mellanox/mlx4/en_ethtool.c), I do not even see the #defines for reporting 56000 links.  ibv_devinfo similarly reports the speed at 40 GBE.

       

      Does something else need to be done to enable 56 GbE support on these cards?

       

      Regards,

      Thomas

        • Re: 56 GbE over a ConnectX-3 VPI adapter
          eddie.notz

          Hi Thomas,

           

          In order to link at 56GbE in a back-2-back setup, you need to change the HCA ini file and re-burn the firmware.

           

          1. Download the .mlx file (fw code file)  and ini file ( configuration file )from:

          Creating a Customized Firmware Binary - Mellanox Technologies -> Firmware DISCLAIMER

           

          2. Extract the files

          3. Find the ini file that belongs to your HCA,

             a. ibv_devinfo | grep board_id

             b. grep <board_id> *

           

          for example:

           

          # tar xzf ConnectX3-rel-2_31_5050.tgz

          # ibv_devinfo | grep board_id

                  board_id:                       MT_1090120019

          # grep MT_1090120019 *

          MCX354A-FCB_A2-A5.ini:PSID = MT_1090120019

           

          4. create a copy of the ini file as a backup

          5. edit the ini file and add the following lines under the [IB] section:

          port1_802_3ap_56kr4_enable = true

          port1_802_3ap_56kr4_ability = true

          port2_802_3ap_56kr4_enable = true

          port2_802_3ap_56kr4_ability = true

           

          6. create a new firmware bin file:

           

          l-supp-11:/tmp/2.31.5050 # mlxburn -fw fw-ConnectX3-rel.mlx -c MCX354A-FCB_A2-A5_56GbE.ini -wrimage fw-ConnectX3-rel-2_31_5050-MCX354A-FCB_A2-A5-56GbE_VPI.bin

          -I- Generating image ...

          -I- Image generation completed successfully.

           

          7. backup the firmware currently loaded on the HCA:

           

          mst start

          flint -d mlx4_0 ri backup.bin

           

          8. burn the new firmware onto the card:

           

          flint -d mlx4_0 -i fw-ConnectX3-rel-2_31_5050-MCX354A-FCB_A2-A5-56GbE_VPI.bin -y b

           

          9. reboot the server

           

          You're done. you can use the same .bin file for the other server if he has the same board_id

           

          Please note that the fw release notes states:

           

          • Added support for 56GbE in all devices supporting 40GbE

          • Establishing 56GbE link with SwitchX® requires 56GbE enablement on the

          switch

          • Establishing 56GbE link on back to back setup requires additions to the INI.

          • For further information, please contact Mellanox Support