In order to link at 56GbE in a back-2-back setup, you need to change the HCA ini file and re-burn the firmware.
1. Download the .mlx file (fw code file) and ini file ( configuration file )from:
2. Extract the files
3. Find the ini file that belongs to your HCA,
a. ibv_devinfo | grep board_id
b. grep <board_id> *
# tar xzf ConnectX3-rel-2_31_5050.tgz
# ibv_devinfo | grep board_id
# grep MT_1090120019 *
MCX354A-FCB_A2-A5.ini:PSID = MT_1090120019
4. create a copy of the ini file as a backup
5. edit the ini file and add the following lines under the [IB] section:
port1_802_3ap_56kr4_enable = true
port1_802_3ap_56kr4_ability = true
port2_802_3ap_56kr4_enable = true
port2_802_3ap_56kr4_ability = true
6. create a new firmware bin file:
l-supp-11:/tmp/2.31.5050 # mlxburn -fw fw-ConnectX3-rel.mlx -c MCX354A-FCB_A2-A5_56GbE.ini -wrimage fw-ConnectX3-rel-2_31_5050-MCX354A-FCB_A2-A5-56GbE_VPI.bin
-I- Generating image ...
-I- Image generation completed successfully.
7. backup the firmware currently loaded on the HCA:
flint -d mlx4_0 ri backup.bin
8. burn the new firmware onto the card:
flint -d mlx4_0 -i fw-ConnectX3-rel-2_31_5050-MCX354A-FCB_A2-A5-56GbE_VPI.bin -y b
9. reboot the server
You're done. you can use the same .bin file for the other server if he has the same board_id
Please note that the fw release notes states:
• Added support for 56GbE in all devices supporting 40GbE
• Establishing 56GbE link with SwitchX® requires 56GbE enablement on the
• Establishing 56GbE link on back to back setup requires additions to the INI.
• For further information, please contact Mellanox Support
Great, thanks for the quick and detailed response! I have created the new firmware, but will wait until the end of the day when the servers are not in use to attempt burning it.
I flashed the cards today and they did establish link at 56 Gbps. They defaulted to IB, but changing the port type to eth worked. I am now seeing ~ 50 Gbps using MPI over RoCE. Thanks again for your help!
when using the last MLNX-OFED and firmware, there is no need to change the ini file (unless you have two servers back to back)
The adapter automatically discovers and changes the speed automatically to 56GbE.
See the flow over here:
for back-2-back 56GbE connections between ConnectX-3 and ConnectX-3 Pro, the solution provided in this post is still relevent