Mellanox Breakout Cables 40G > 4x10G and 100G > 4x25G

Version 24

    This post is focused on Mellanox breakout cables, usage and configuration options.

     

    copper_splitter_QSFP_to_4xSFP .jpg

    References

     

    Note:

    • For 100GE switches, Mellanox offers a passive copper hybrid cable, ETH 100GbE to 4x25GbE, QSFP28 to 4xSFP28. P/N MCP7F00-A0xx. There are several lengths available. This post shows the procedure for 40Gb/E split to 4x10GbE, but the procedure is similar for 100GbE hybrid cables. This cable is supports also 40GbE > 10GbE, as it is backwards compatible.
    • For 40GE switches, Mellanox offers a passive copper hybrid cable, ETH 40GbE to 4x10GbE, QSFP to 4xSFP+. P/N MC2609130-00./5 or MC2309124-xxx.

     

    In case you have Cumulus Spectrum switch, see HowTo Configure Breakout Cables 40G > 4x10G using Cumulus Linux.

     

     

    Usage and Configuration Videos - Mellanox Academy

    The following Academy videos explains about the Usage and Configuration of Breakout Cables of 40GbE -> 4x10GbE and 100GbE -> 25GbE, with an example of Spectrum Switch.

     

     

     

    Setup

    Mellanox breakout cable P/Ns (passive copper):

    • MC2609130-001 (1m)
    • MC2609130-003 (3m)
    • MC2609125-005 (5m)

     

    Mellanox Ethernet switch systems (Ethernet Switch Systems - Mellanox Technologies):

    • SwitchX IC based Switches
      • SX1012
      • SX1024
      • SX1036
      • SX1410
      • SX1710
    • Spectrum IC based Switches
      • SN2700
      • SN2410
      • SN2100

     

    Note: It will also possible to work with Mellanox VPI switches, equipped with gateway license (SX6012, SX6018, SX6036).

     

    Overview

    The Breakout cable is a unique Mellanox capability, where a single physical 40GbE port is divided into 4x10GbE (or 2x10GbE) NIC ports. It maximizes the flexibility of the end user to use the Mellanox switch with a combination of 10Gbps and 40Gbps interfaces according to the specific requirements of its network.

    Important: The 40GbE split options is supported only on Mellanox switches and not supported on Mellanox adapters (e.g. ConnectX-3) when equipped with 40GbE ports. In case you wish to limit the 40GbE port on the adapter to 10GbE you can use QSA or similar 40GbE-10GbE cables (Refer to Mellanox.com - here)

     

    Split options:

    Not all 40GbE switch ports can be split to 10GbE ports, certain ports cannot be split at all and there are 40GbE ports which can be split into 2x10GbE ports only. The ports that can be split into 2x10GbE ports, using lanes (connectors) 1 and 2 only. Connectors 3 and 4 are left unconnected. In addition, when performing the split, additional port may be unmapped and cannot be used. The map of ports that can be split per switch can be found in the Switch Hardware Manual located in Mellanox website.

    Note: The maximum number of ports (mixture of 40GbE and 10GbE) is 64 ports.


    Interface Indexing on the switch:

    Splitting a 40GbE switch port changes the notation of that port in the CLI from x/y to x/y/z while “x/y” indicating the previous notation of the port prior to the split and “z” indicating the number of the resulting 10G port (1,2 or 1,2,3,4). Each sub-physical port is then handled as an individual port.

     

    For example, in case of SX1036 switch:

    • Port 1/1 get unmapped in case port 1/4 is split.
    • Splitting port 1/4 into 4 ports will give the following new ports: 1/4/1, 1/4/2, 1/4/3, 1/4/4.
    • Before the split the port will have the notation of ethernet 1/4, while after the split, the port ethernet 1/1 will be removed (unmapped) and 4 new 10GbE ports will be added to the switch:
      • ethernet 1/4/1
      • ethernet 1/4/2
      • ethernet 1/4/3
      • ethernet 1/4/4

     

    Configuration

    To spit a 40GbE port to 4x10GbE port follow the next steps:

    (1) Connect the breakout cable on the 40GbE side to a port in Mellanox switch that can be split (follow Mellanox switch hardware manual to verify which port can be split).

    (2) Connect the breakout cable on the 4x10GbE side to 4x10GbE adapters (e.g. ConnectX-3)

    (3) Login to Mellanox switch (e.g. SX1036) and run the following commands (assuming the port to be split is "ethernet 1/4"):

         - Shutdown all related interfaces to the port to be split. In SX1036 case port number 1/1 is attached to port number 1/4. When performing the split, both ports will be unmapped.

    switch (config) # interface ethernet 1/4 shutdown

    switch (config) # interface ethernet 1/1 shutdown

        

     

         - Split the port

    switch (config) # interface ethernet 1/4 module-type qsfp-split-4 force

    the following interfaces are being unmapped: 1/4 1/1

    switch (config) #

         

     

    Note: in case of SX1012 there is no dependency. Simply shutdown each port and then change the module-type. For example, split port 1/4 on SX1012:

    switch (config) # interface ethernet 1/4 shutdown

    switch (config) # interface ethernet 1/4 module-type qsfp-split-4 force

     

    As for the other switches, refer to the Hardware User Manual (located on Mellanox.com - per switch) or MLNX-OS User Manual to see the port-map to understand which ports can be split and how.

     

    Now you can start configuring the ports 1/4/1, 1/4/2, 1/4/3, 1/4/4 as needed.

     

    If you wish to un-split 4x10GbE ports run the following commands:

    switch (config) # interface ethernet 1/4/1  shutdown

    switch (config) # interface ethernet 1/4/2  shutdown

    switch (config) # interface ethernet 1/4/3  shutdown

    switch (config) # interface ethernet 1/4/4  shutdown

    switch  (config) # interface ethernet 1/4/1  module-type qsfp force

    the following interfaces are being unmapped: 1/4/1 1/4/2 1/4/3 1/4/4

    switch (config) #