Rack Solution Using SN2100 MLAG Switch Pair and ConnectX-4 Lx

Version 9

    This post discusses a simple rack solution using two Mellanox SN2100 (16 100GbE ports) switches and up to 48 servers installed with ConnectX-4 Lx. The post assumes that the reader knows what  MLAG is, and understands MLAG configuration in general. The post does not explain the theory behind MLAG and breakout cables, but only describes the running config needed. To learn more about Mellanox Ethernet Switches and MLAG, refer to Ethernet Switch Solutions.

     

    References

     

    BOM

    Product
    DescriptionQuantity
    SN2100 100GbE switch16x100GbE Ethernet Spectrum Switch 1/2 width2
    100GbE => 4x25GbE breakout cableSplit cable to be used on 12 ports12
    100GbE copper cablesTo be used for the IPL link2
    40/100GbE optical cables (or transceivers)To be used for uplinks4
    ConnectX-4 Lx dual port2x25GbE port adapter (SFP28)48

     

    Setup

     

    High-level Plan

    Switch Design and Connectivity

    • Two SN2100 will be configured as MLAG switch pair for redundancy

    • Ports 1/15-1/16 of each switch will be used for the IPL link (MLAG control protocol)
    • Ports 1/13-1/14 of each switch will be used for uplinks
    • Ports 1/1-1/12 will be used to connect to the servers via 100GbE > 25GbE breakout cable (giving 48 links of 25GbE)

     

    Server Design and Connectivity

    • Up to 48 servers
    • Each server could be installed with a ConnectX-4 Lx adapter (25G) dual port

     

    Management Network

    • 1Gb out-of-band management network should be supplied for this design, the configuration is out of scope for this design
    • For the configuration example, we will assume the management IP addresses of the switches are as follows:
      • Left switch: 10.10.10.1
      • Right switch: 10.10.10.2
      • MLAG virtual IP (VIP): 10.10.10.3

     

    Configuration

    Switch Configuration

    Review the following posts to understand MLAG configuration and breakout cable procedures:

     

    Here is the basic configuration (running-config) example for this setup. The configuration could be updated according to the specific setup/application.

    Note: VLANs were not configured here (other than VLAN 2 for the IPL link).

     

    ConfigurationLeft SN2100 SwitchRight SN2100 Switch
    MLAG switch setup (IPL ports)

    protocol mlag

    interface port-channel 1

    interface ethernet 1/15-1/16 channel-group 1 mode active

    lacp

    vlan 2

    no spanning-tree

    ip routing vrf default

    interface vlan 2

    interface vlan 2 ip address 1.1.1.1 255.255.255.252

    dcb priority-flow-control enable force

    lldp

    mlag-vip mlag-cluster ip 10.10.10.3/24 force

    no mlag shutdown

    mlag system-mac AA:AA:AA:AA:AA:AA

    interface port-channel 1 ipl 1

    interface vlan 2 ipl 1 peer-address 1.1.1.2

    interface port-channel 1 dcb priority-flow-control mode on force

    protocol mlag

    interface port-channel 1

    interface ethernet 1/15-1/16 channel-group 1 mode active

    lacp

    vlan 2

    no spanning-tree

    ip routing vrf default

    interface vlan 2

    interface vlan 2 ip address 1.1.1.2 255.255.255.252

    dcb priority-flow-control enable force

    lldp

    mlag-vip mlag-cluster ip 10.10.10.3/24 force

    no mlag shutdown

    mlag system-mac AA:AA:AA:AA:AA:AA

    interface port-channel 1 ipl 1

    interface vlan 2 ipl 1 peer-address 1.1.1.1

    interface port-channel 1 dcb priority-flow-control mode on force

    Breakout cable ports 1-12

    interface ethernet 1/1 module-type qsfp-split-4 force

    interface ethernet 1/2 module-type qsfp-split-4 force

    interface ethernet 1/3 module-type qsfp-split-4 force

    interface ethernet 1/4 module-type qsfp-split-4 force

    interface ethernet 1/5 module-type qsfp-split-4 force

    interface ethernet 1/6 module-type qsfp-split-4 force

    interface ethernet 1/7 module-type qsfp-split-4 force

    interface ethernet 1/8 module-type qsfp-split-4 force

    interface ethernet 1/9 module-type qsfp-split-4 force

    interface ethernet 1/10 module-type qsfp-split-4 force

    interface ethernet 1/11 module-type qsfp-split-4 force

    interface ethernet 1/12 module-type qsfp-split-4 force

    interface ethernet 1/1 module-type qsfp-split-4 force

    interface ethernet 1/2 module-type qsfp-split-4 force

    interface ethernet 1/3 module-type qsfp-split-4 force

    interface ethernet 1/4 module-type qsfp-split-4 force

    interface ethernet 1/5 module-type qsfp-split-4 force

    interface ethernet 1/6 module-type qsfp-split-4 force

    interface ethernet 1/7 module-type qsfp-split-4 force

    interface ethernet 1/8 module-type qsfp-split-4 force

    interface ethernet 1/9 module-type qsfp-split-4 force

    interface ethernet 1/10 module-type qsfp-split-4 force

    interface ethernet 1/11 module-type qsfp-split-4 force

    interface ethernet 1/12 module-type qsfp-split-4 force

    MLAG port-channel interfaces 1-12

    interface mlag-port-channel 1-48

     

    interface ethernet 1/1/1 mlag-channel-group 1 mode on

    interface ethernet 1/1/2 mlag-channel-group 2 mode on

    interface ethernet 1/1/3 mlag-channel-group 3 mode on

    interface ethernet 1/1/4 mlag-channel-group 4 mode on

    interface ethernet 1/2/1 mlag-channel-group 5 mode on

    interface ethernet 1/2/2 mlag-channel-group 6 mode on

    interface ethernet 1/2/3 mlag-channel-group 7 mode on

    interface ethernet 1/2/4 mlag-channel-group 8 mode on

    ...

    interface ethernet 1/12/1 mlag-channel-group 45 mode on

    interface ethernet 1/12/2 mlag-channel-group 46 mode on

    interface ethernet 1/12/3 mlag-channel-group 47 mode on

    interface ethernet 1/12/4 mlag-channel-group 48 mode on

     

    interface mlag-port-channel 1-48 no shutdown

    interface mlag-port-channel 1-48

     

    interface ethernet 1/1/1 mlag-channel-group 1 mode on

    interface ethernet 1/1/2 mlag-channel-group 2 mode on

    interface ethernet 1/1/3 mlag-channel-group 3 mode on

    interface ethernet 1/1/4 mlag-channel-group 4 mode on

    interface ethernet 1/2/1 mlag-channel-group 5 mode on

    interface ethernet 1/2/2 mlag-channel-group 6 mode on

    interface ethernet 1/2/3 mlag-channel-group 7 mode on

    interface ethernet 1/2/4 mlag-channel-group 8 mode on

    ...

    interface ethernet 1/12/1 mlag-channel-group 45 mode on

    interface ethernet 1/12/2 mlag-channel-group 46 mode on

    interface ethernet 1/12/3 mlag-channel-group 47 mode on

    interface ethernet 1/12/4 mlag-channel-group 48 mode on

     

    interface mlag-port-channel 1-48 no shutdown

     

    Server Configuration

    • Install the OS and latest driver and firmware
    • Configure LAG (bond) between the two adapter ports on each of the servers.