This post is archived.
This post is showing how to configure a link to reach 56GbE between Mellanox adapters and switches.
- HowTo Setup 56GbE Back-to-Back on two servers
- Change Mellanox 40GbE to 56GbE with RoCE over Windows Servers
(1) Make sure the host is equipped with Mellanox ConnectX-3 adapter, and installed with the latest MLNX-OFED release.
(2) 56GbE on Mellanox switch systems requires a license key, make sure you get a license from Mellanox representatives. 40-56GbE Switch Upgrade License is Now Free , contact email@example.com to receive it.
(3) Make sure you use Mellanox cable that supports 56Gb/s refer to (for example, use MC2207130-001)
Note: there is no need to change anything on the host side, when using the latest firmware.
Follow the next steps to reach 56GbE on a link between Mellanox adapter and a switch.
(1) Connect Mellanox ConnectX-3 adapter to one of the Mellanox switch ports with a 56Gb/s capable cable.
(2) Install the 56GbE license on Mellanox switch.
switch (config) # license install LK2-EFM_SX-LOP3-0KAL-64RK-8CTQ-60R3-0C1L-5R12-5L11-7
License was installed successfully. Please wait 1 minute before further configurations.
(3) Change the speed on the relevant port connected to the adapter (e.g. eth 1/1)
switch (config) # interface ethernet 1/1 speed 56000 force
(4) Restart the network driver on the server
# /etc/init.d/openibd restart
(5) To verify, check the port status and link speed on the driver and on the switch
# show interfaces ethernet 1/1
Admin state: Enabled
Operational state: Up
Mac address: f4:52:14:11:e4:2d
MTU: 1500 bytes(Maximum packet size 1522 bytes)
Flow-control: receive off send off
Actual speed: 56 Gbps
Width reduction mode: disabled
Switchport mode: access
Last clearing of "show interface" counters : 6D:21H
60 seconds ingress rate: 0 bits/sec, 0 bytes/sec, 0 packets/sec
60 seconds egress rate: 264 bits/sec, 33 bytes/sec, 1 packets/sec
19 unicast packets
30 multicast packets
13 broadcast packets
0 error packets
0 discard packets
34 unicast packets
294879 multicast packets
0 broadcast packets
2 discard packets
r-qa-sit-sx01 [standalone: master] #
CA type: MT4103
Number of ports: 2
Firmware version: 2.31.6244
Hardware version: 0
Node GUID: 0xf452140300171b80
System image GUID: 0xf452140300171b83
Physical state: LinkUp
Base lid: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0xf65214fffe171b81
Link layer: Ethernet
Physical state: Polling
Base lid: 0
SM lid: 0
Capability mask: 0x02514868
Port GUID: 0xf452140300171b82
Link layer: InfiniBand