This post describes how to configure breakout cable 40 => 10Gb/s using Cumulus Linux OS.
This post assumes that you already have Cumulus Linux installed on a Mellanox Spectrum™ switch.
- HowTo Install Cumulus Linux OS Over ONIE on Mellanox Ethernet Switches
- HowTo Configure Switch Interface Using Cumulus Linux
- Mellanox Breakout Cables 40G > 4x10G and 100G > 4x25G
- HowTo Configure VLANs on Cumulus Linux
- Layer 1 and Switch Port Attributes - Cumulus Linux 3.3.1 - Cumulus Networks
1. Two servers connected to a Cumulus Linux switch. In this example we will split switch port 5 (swp5) to 4x10G (swps0, swps1, swps2, swps3) while disabling port 6 (swp6).
2. Used breakout cable PN: MC2609130
Make sure to have a valid licence installed for Cumulus Linux.
1. Log into the switch using the following credentials:
2. Switch to root user. Run sudo -i.
# sudo -i
[sudo] password for cumulus:
Please send these support file(s) to firstname.lastname@example.org:
3. Set the port splitting using the /etc/cumulus/ports.conf configuration file. Open the file and edit swp5 and swp6 as follows.
5=4x10G --> Splits swp5
6=disabled --> Deletes swp6
Note: When you split one port you must delete another.
Note: Refer to the switch HW user manual (on Mellanox.com) to understand which port to delete. In case of SN2100, the mapping is different from SN2700 for example.
The full /etc/cumulus/ports.conf file is presented in what follows:
# ports.conf --
# This file controls port speed, aggregation and subdivision.
# For example, the QSFP28 ports can be split into multiple interfaces. This
# file sets the number of interfaces per port and the speed of those interfaces.
# You must restart switchd for changes to take effect.
# mlnx,x86_MSN2700 has:
# 32 QSFP28 ports numbered 1-32
# These ports are configurable as 40G, 50G, 2x50G, or 100G; or a subset
# of them can be split into 4x25G or 4x10G.
# Note that a port may become disabled, i.e., unusable and unconfigurable
# in /etc/network/interfaces, when an adjacent port is split into 4
# interfaces. It is REQUIRED that the disabled port be configured as
# "disabled" in this file when an adjacent port is split into 4
# NOTE: When ports are split into 4 interfaces it is REQUIRED that the adjacent
# disabled port be configured as "disabled" in this file. When splitting a port
# into two interfaces, like 2x50G, it is NOT required that the adjacent port be
# disabled. Adjacent ports only need to be disabled when a port is split into
# 4 interfaces. For example, when splitting port 11 into 4 25G interfaces, port
# 12 must be configued as "disabled" like this:
# The list of ports which can be split into 4 interfaces and the adjacent ports
# which must be configured as "disabled" are:
# 1: 4x10G or 4x25G (would disable port 2)
# 3: 4x10G or 4x25G (would disable port 4)
# 5: 4x10G or 4x25G (would disable port 6)
# 7: 4x10G or 4x25G (would disable port 8)
# 9: 4x10G or 4x25G (would disable port 10)
# 11: 4x10G or 4x25G (would disable port 12)
# 13: 4x10G or 4x25G (would disable port 14)
# 15: 4x10G or 4x25G (would disable port 16)
# 17: 4x10G or 4x25G (would disable port 18)
# 19: 4x10G or 4x25G (would disable port 20)
# 21: 4x10G or 4x25G (would disable port 22)
# 23: 4x10G or 4x25G (would disable port 24)
# 25: 4x10G or 4x25G (would disable port 26)
# 27: 4x10G or 4x25G (would disable port 28)
# 29: 4x10G or 4x25G (would disable port 30)
# 31: 4x10G or 4x25G (would disable port 32)
# QSFP28 ports
# <port label> = [40G|50G|100G]
# or when split = [2x50G|4x10G|4x25G|disabled]
4. Open /etc/network/interfaces, and create a new port section for the relevant ports. In this case, create the 4 new ports in the file:
5. Apply the configuration. Run:
# ifreload -a
6. Check the link status. Run:
# netshow interface
Name Speed MTU Mode Summary
----- ------ ---------- ----- ------------- ------------------------
UP lo N/A 65536 Loopback IP: 127.0.0.1/8, ::1/128
UP eth0 1G 1500 Mgmt IP: 10.20.2.70/16(DHCP)
ADMDN eth1 0M 1500 Mgmt
UP swp5s0 10G(4x10G) 1500 NotConfigured
UP swp5s1 10G(4x10G) 1500 NotConfigured
UP swp5s2 10G(4x10G) 1500 NotConfigured
UP swp5s3 10G(4x10G) 1500 NotConfigured
You can see that all split ports are UP.
Configuration using NCLU
Starting with Cumulus Linux v3.2.1 you can use NCLU (Cumulus CLI) to configure breakout cables.
For example in order to split port swp5 to 4x and to disable port swp6 run:
# net add interface swp1 breakout
1x : Do not split or gang the port
2x : Split the port into two interfaces
4x : Split the port into four interfaces
/4 : Gang four ports into a single port
disabled : Disabled
# net add interface swp5 breakout 4x
# net add interface swp6 breakout disabled
# net commit
1. When changing the speed to 10GbE, you might need to force the speed on the server side as well (if the adapter is not a 10GbE adapter). See HowTo Change the Ethernet Port Speed of Mellanox Adapters (Linux) for more information.
This is an example with ConnectX®-4 100GbE QSFP using QSA to lower the speed to 10GbE.
Forcing the port on the server to be at the speed 10000 with autoneg off may not work.
ethtool -s ens785f1 autoneg off speed 10000
Run the following on the server side on the link connected on the breakout cable (swp5s0) to change the autoneg to on.
# ethtool -s ens785f1 autoneg on
2. Restart the driver to make sure that the configuration gets loaded.
# systemctl restart switchd.service