HowTo Setup 56GbE Back-to-Back on two servers

Version 3

    This post is archived.

     

    This post takes you through the procedure to enable 56GbE back-to-back (Two servers connected to each other).

    This post is advanced and includes firmware burning.

     

    References

     

    Setup

    The procedure below was used on two servers equipped with ConnectX-3 Pro adapter card connected back to back with 56Gb/s cable.

     

    Port speed is currently 40GbE:

    # ibdev2netdev

    mlx4_0 port 1 ==> enp2s0 (Up)

    mlx4_0 port 2 ==> enp2s0d1 (Down)

     

    # ethtool enp2s0

    Settings for enp2s0:

      Supported ports: [ FIBRE ]

      Supported link modes:   1000baseKX/Full

                             10000baseKX4/Full

                             40000baseCR4/Full

                             40000baseSR4/Full

      Supported pause frame use: Symmetric Receive-only

      Supports auto-negotiation: Yes

      Advertised link modes:  1000baseKX/Full

                             10000baseKX4/Full

                             40000baseCR4/Full

                             40000baseSR4/Full

      Advertised pause frame use: Symmetric

      Advertised auto-negotiation: Yes

      Link partner advertised link modes:  40000baseCR4/Full

      Link partner advertised pause frame use: No

      Link partner advertised auto-negotiation: Yes

      Speed: 40000Mb/s

      Duplex: Full

      Port: Direct Attach Copper

      PHYAD: 0

      Transceiver: internal

      Auto-negotiation: on

      Supports Wake-on: d

      Wake-on: d

      Current message level: 0x00000014 (20)

            link ifdown

      Link detected: yes

     

    Procedure

    1. Download the tarball firmware for the specific card from http://www.mellanox.com/page/custom_firmware_table

     

    For example:

    # cd /tmp

    # wget http://www.mellanox.com/downloads/firmware/ConnectX3Pro-rel-2_34_5000-web.tgz .

    ...

     

    2. Extract the files.

    # tar -xzvf ConnectX3Pro-rel-2_34_5000-web.tgz

    ./

    ./MCX312C-XCC_Ax.ini

    ./MCX342A-XCQ_Ax.ini

    ./MCX312B-XCC_Ax.ini

    ./MCX314A-BCC_Ax.ini

    ./MCX341A-XCP_Ax.ini

    ./MCX366A-BCC_Ax.ini

    ./MCX354A-FCC_Ax.ini

    ./MCX311A-XCC_Ax.ini

    ./MCX342A-XCP_Ax.ini

    ./MCX341A-XCQ_Ax.ini

    ./MCX353A-FCC_Ax.ini

    ./MCX313A-BCC_Ax.ini

    ./fw-ConnectX3Pro-rel.mlx

     

    3. Find the specific ini file that belong to your adapter.

    # ibv_devinfo | grep board_id

      board_id: MT_1090111019

     

    # grep MT_1090111019 *

    MCX354A-FCC_Ax.ini:PSID = MT_1090111019

     

    The ini file to be used it MCX354A-FCC_Ax.ini

     

    4. Create a copy of the ini file as backup.

    # cp MCX354A-FCC_Ax.ini MCX354A-FCC_Ax.ini.org

     

    5. Edit following entries in the ini file, under [IB] section.

    port1_802_3ap_56kr4_ability = true

    port2_802_3ap_56kr4_ability = true

    port1_802_3ap_56kr4_enable = true

    port2_802_3ap_56kr4_enable = true

     

    6. Create a new firmware bin file (using the mlx file and the new ini file).

    # mlxburn -fw fw-ConnectX3Pro-rel.mlx -c MCX354A-FCC_Ax.ini -wrimage fw-ConnectX3Pro-rel-2_34_5000-MCX354A-FCC_Ax.bin

    -I- Generating image ...

    -I- Image generation completed successfully.

     

    7. Backup the firmware image currently loaded to the adapter.

    # mst start

    # flint -d mlx4_0 ri backup.bin

     

    8. Burn the new bin firmware file info the adapter card.

    # flint -d mlx4_0 -i fw-ConnectX3Pro-rel-2_34_5000-MCX354A-FCC_Ax.bin -y b

     

        Current FW version on flash:  2.33.5100

        New FW version:               2.34.5000

     

    Burning FS2 FW image without signatures - OK 

    Restoring signature                     - OK

     

    9. Reboot the server.

     

    10. Run the same procedure on the other server.

     

    Note: you can use the same bin file for the other server in case it has the same board id.

     

    11. Check the link speed.

    # ibdev2netdev

    mlx4_0 port 1 ==> enp2s0 (Up)

    mlx4_0 port 2 ==> ib0 (Down)

     

    # ethtool enp2s0

    Settings for enp2s0:

      Supported ports: [ FIBRE ]

      Supported link modes:   1000baseKX/Full

                             10000baseKX4/Full

                             40000baseCR4/Full

                             40000baseSR4/Full

      Supported pause frame use: Symmetric Receive-only

      Supports auto-negotiation: Yes

      Advertised link modes:  1000baseKX/Full

                             10000baseKX4/Full

                             40000baseCR4/Full

                             40000baseSR4/Full

      Advertised pause frame use: Symmetric

      Advertised auto-negotiation: Yes

      Link partner advertised link modes:  40000baseCR4/Full

      Link partner advertised pause frame use: No

      Link partner advertised auto-negotiation: Yes

      Speed: 56000Mb/s

      Duplex: Full

      Port: Direct Attach Copper

      PHYAD: 0

      Transceiver: internal

      Auto-negotiation: on

      Supports Wake-on: d

      Wake-on: d

      Current message level: 0x00000014 (20)

            link ifdown

      Link detected: yes