HowTo Configure Breakout Cables 40G => 4x10G Using Cumulus Linux

Version 13

    This post describes how to configure breakout cable 40 => 10Gb/s using Cumulus Linux OS.

    This post assumes that you already have Cumulus Linux installed on a Mellanox Spectrum™ switch.

     

    References

     

    Setup

    1. Two servers connected to a Cumulus Linux switch. In this example we will split switch port 5 (swp5) to 4x10G (swps0, swps1, swps2, swps3) while disabling port 6 (swp6).

     

     

    2. Used breakout cable PN: MC2609130

     

    Make sure to have a valid licence installed for Cumulus Linux.

     

    Configuration

    1. Log into the switch using the following credentials:

    Username: cumulus

    Password: CumulusLinux!

     

    2. Switch to root user. Run sudo -i.

    Password: CumulusLinux!

     

    # sudo -i

    [sudo] password for cumulus:

     

    *****************************************************************

    Please send these support file(s) to support@cumulusnetworks.com:

    /var/support/cl_support__cumulus_20161018_211850.tar.xz

    /var/support/cl_support__cumulus_20161018_212948.tar.xz

    /var/support/cl_support__cumulus_20161018_213430.tar.xz

    /var/support/cl_support__cumulus_20161018_213607.tar.xz

    /var/support/cl_support__cumulus_20161018_213758.tar.xz

    /var/support/cl_support__cumulus_20161018_213924.tar.xz

    /var/support/cl_support__cumulus_20161018_220157.tar.xz

    *****************************************************************

     

    3. Set the port splitting using the /etc/cumulus/ports.conf configuration file. Open the file and edit swp5 and swp6 as follows.

    5=4x10G      --> Splits swp5

    6=disabled   --> Deletes swp6

     

    Note: When you split one port you must delete another.

     

    Note: Refer to the switch HW user manual (on Mellanox.com) to understand which port to delete. In case of SN2100, the mapping is different from SN2700 for example.

     

    The full /etc/cumulus/ports.conf file is presented in what follows:

     

    # ports.conf --

    #

    #        This file controls port speed, aggregation and subdivision.

    #

    # For example, the QSFP28 ports can be split into multiple interfaces. This

    # file sets the number of interfaces per port and the speed of those interfaces.

    #

    # You must restart switchd for changes to take effect.

    #

    # mlnx,x86_MSN2700 has:

    #     32 QSFP28 ports numbered 1-32

    #         These ports are configurable as 40G, 50G, 2x50G, or 100G; or a subset

    #         of them can be split into 4x25G or 4x10G.

    #

    #         Note that a port may become disabled, i.e., unusable and unconfigurable

    #         in /etc/network/interfaces, when an adjacent port is split into 4

    #         interfaces. It is REQUIRED that the disabled port be configured as

    #         "disabled" in this file when an adjacent port is split into 4

    #         interfaces.

    #

    # NOTE: When ports are split into 4 interfaces it is REQUIRED that the adjacent

    # disabled port be configured as "disabled" in this file. When splitting a port

    # into two interfaces, like 2x50G, it is NOT required that the adjacent port be

    # disabled. Adjacent ports only need to be disabled when a port is split into

    # 4 interfaces. For example, when splitting port 11 into 4 25G interfaces, port

    # 12 must be configued as "disabled" like this:

    #

    #   11=4x25G

    #   12=disabled

    #

    #  The list of ports which can be split into 4 interfaces and the adjacent ports

    #  which must be configured as "disabled" are:

    #

    #  1:  4x10G or 4x25G (would disable port 2)

    #  3:  4x10G or 4x25G (would disable port 4)

    #  5:  4x10G or 4x25G (would disable port 6)

    #  7:  4x10G or 4x25G (would disable port 8)

    #  9:  4x10G or 4x25G (would disable port 10)

    #  11: 4x10G or 4x25G (would disable port 12)

    #  13: 4x10G or 4x25G (would disable port 14)

    #  15: 4x10G or 4x25G (would disable port 16)

    #  17: 4x10G or 4x25G (would disable port 18)

    #  19: 4x10G or 4x25G (would disable port 20)

    #  21: 4x10G or 4x25G (would disable port 22)

    #  23: 4x10G or 4x25G (would disable port 24)

    #  25: 4x10G or 4x25G (would disable port 26)

    #  27: 4x10G or 4x25G (would disable port 28)

    #  29: 4x10G or 4x25G (would disable port 30)

    #  31: 4x10G or 4x25G (would disable port 32)

    #

    # QSFP28 ports

    #

    # <port label>    = [40G|50G|100G]

    #   or when split = [2x50G|4x10G|4x25G|disabled]

    1=100G

    2=100G

    3=100G

    4=100G

    5=4x10G

    6=disabled

    7=100G

    8=100G

    9=100G

    10=100G

    11=100G

    12=100G

    13=100G

    14=100G

    15=100G

    16=100G

    17=100G

    18=100G

    19=100G

    20=100G

    21=100G

    22=100G

    23=100G

    24=100G

    25=100G

    26=100G

    27=100G

    28=100G

    29=100G

    30=100G

    31=100G

    32=100G

     

    4. Open /etc/network/interfaces, and create a new port section for the relevant ports. In this case, create the 4 new ports in the file:

    • swps0
    • swps1
    • swps2
    • swps3

     

    For example:

    #swp5s0

    auto swp5s0

    iface swp5s0

      link-speed 10000

      mtu 1500

     

    #swp5s1

    auto swp5s1

    iface swp5s1

      link-speed 10000

      mtu 1500

     

    #swp5s2

    auto swp5s2

    iface swp5s2

      link-speed 10000

      mtu 1500

     

    #swp5s3

    auto swp5s3

    iface swp5s3

      link-speed 10000

      mtu 1500

     

     

    5. Apply the configuration. Run:

    # ifreload -a

     

    6. Check the link status. Run:

    # netshow interface

     

           Name    Speed         MTU  Mode           Summary

    -----  ------  ----------  -----  -------------  ------------------------

    UP     lo      N/A         65536  Loopback       IP: 127.0.0.1/8, ::1/128

    UP     eth0    1G           1500  Mgmt           IP: 10.20.2.70/16(DHCP)

    ADMDN  eth1    0M           1500  Mgmt

    UP     swp5s0  10G(4x10G)   1500  NotConfigured

    UP     swp5s1  10G(4x10G)   1500  NotConfigured

    UP     swp5s2  10G(4x10G)   1500  NotConfigured

    UP     swp5s3  10G(4x10G)   1500  NotConfigured

     

    You can see that all split ports are UP.

     

    Configuration using NCLU

    Starting with  Cumulus Linux v3.2.1 you can use NCLU (Cumulus CLI) to configure breakout cables.

     

    For example in order to split port swp5 to 4x and to disable port swp6 run:

    # net add interface swp1 breakout

        1x        :  Do not split or gang the port

        2x        :  Split the port into two interfaces

        4x        :  Split the port into four interfaces

        /4        :  Gang four ports into a single port

        disabled  :  Disabled

     

     

    # net add interface swp5 breakout 4x

    # net add interface swp6 breakout disabled

    # net commit

     

    Troubleshooting

    1.  When changing the speed to 10GbE, you might need to force the speed on the server side as well (if the adapter is not a 10GbE adapter). See HowTo Change the Ethernet Port Speed of Mellanox Adapters (Linux) for more information.

     

    This is an example with ConnectX®-4 100GbE QSFP using QSA to lower the speed to 10GbE.

     

    Forcing the port on the server to be at the speed 10000 with autoneg off may not work.

    ethtool  -s ens785f1  autoneg off speed 10000

     

    Run the following on the server side on the link connected on the breakout cable (swp5s0) to change the autoneg to on.

    # ethtool -s ens785f1 autoneg on

     

    2. Restart the driver to make sure that the configuration gets loaded.

    # systemctl restart switchd.service