HowTo Configure 56GbE Link on Mellanox Adapters and Switches

Version 11

    This post is archived.

     

    This post is showing how to configure a link to reach 56GbE between Mellanox adapters and switches.

     

    References

     

    Prerequisite

    (1) Make sure the host is equipped with Mellanox ConnectX-3 adapter, and installed with the latest MLNX-OFED release.

    (2) 56GbE on Mellanox switch systems requires a license key, make sure you get a license from Mellanox representatives. 40-56GbE Switch Upgrade License is Now Free , contact support@mellanox.com to receive it.

    (3) Make sure you use Mellanox cable that supports 56Gb/s refer to (for example, use MC2207130-001)

     

    Note: there is no need to change anything on the host side, when using the latest firmware.

    Follow the next steps to reach 56GbE on a link between Mellanox adapter and a switch.

    (1) Connect Mellanox ConnectX-3 adapter to one of the Mellanox switch ports with a 56Gb/s capable cable.

     

    (2) Install the 56GbE license on Mellanox switch.

    switch (config) # license install LK2-EFM_SX-LOP3-0KAL-64RK-8CTQ-60R3-0C1L-5R12-5L11-7

    License was installed successfully. Please wait 1 minute before further configurations.

         

     

    (3) Change the speed on the relevant port connected to the adapter (e.g. eth 1/1)

    switch (config) # interface ethernet 1/1 speed 56000 force

         

     

    (4) Restart the network driver on the server

    # /etc/init.d/openibd restart

         

     

    (5) To verify, check the port status and link speed on the driver and on the switch

    # show interfaces ethernet 1/1

    Eth1/1

      Admin state: Enabled
      Operational state: Up
      Description: N\A
      Mac address: f4:52:14:11:e4:2d
      MTU: 1500 bytes(Maximum packet size 1522 bytes)
      Flow-control: receive off send off
      Actual speed: 56 Gbps
      Width reduction mode: disabled
      Switchport mode: access
      Last clearing of "show interface" counters : 6D:21H
      60 seconds ingress rate: 0 bits/sec, 0 bytes/sec, 0 packets/sec
      60 seconds egress rate: 264 bits/sec, 33 bytes/sec, 1 packets/sec

    Rx
      62                   packets
      19                   unicast packets
      30                   multicast packets
      13                   broadcast packets
      9751                 bytes
      0                    error packets
      0                    discard packets

    Tx
      294913               packets
      34                   unicast packets
      294879               multicast packets
      0                    broadcast packets
      18921528             bytes
      2                    discard packets

    r-qa-sit-sx01 [standalone: master] #

         

     

    # ibstat

    CA 'mlx4_0'

            CA type: MT4103

            Number of ports: 2

            Firmware version: 2.31.6244

            Hardware version: 0

            Node GUID: 0xf452140300171b80

            System image GUID: 0xf452140300171b83

            Port 1:

                    State: Active

                    Physical state: LinkUp

                    Rate: 56

                    Base lid: 0

                    LMC: 0

                    SM lid: 0

                    Capability mask: 0x00010000

                    Port GUID: 0xf65214fffe171b81

                    Link layer: Ethernet

            Port 2:

                    State: Down

                    Physical state: Polling

                    Rate: 10

                    Base lid: 0

                    LMC: 0

                    SM lid: 0

                    Capability mask: 0x02514868

                    Port GUID: 0xf452140300171b82

                    Link layer: InfiniBand