Single Rack HA Layer 2 MAGP network deployment with Mellanox  NEO™

Version 22

    This post describes procedure to configure Single Rack High Availability Layer 2 (No LACP) network deployment with with Mellanox NEO™ for Enterprise Data Centers.

    Introduction

    As new applications are constantly evolving, data centers must be flexible and future proof to meet the demand for higher throughput.

    Traditionally, aggregation switching in data centers of small deployment has been based on modular switches.

    The Mellanox Virtual Modular Switch® solution (VMS) provides an ideal, optimized approach for one rack.

    Configuration of the VMS infrastructure is performed by the VMS Wizard via Mellanox NEO UI.

    In this document the Mellanox NEO™ is used to configure L2 Ethernet network on two Top of the Rack (TOR) switches also known as Leaf switches.

    There are two additional ways to do that:

    • CLI based configuration done one-by-one on all switches - see following link
    • Wizard based configuration with Ansible - see following link

     

    Related References

     

    Mellanox components in the Architecture

    The Mellanox NEO™ is a powerful platform for managing computing networks. It enables data center operators to efficiently provision, monitor and operate the modern data center fabric.

    The Mellanox Spectrum switch family provides the most efficient network solution for the ever-increasing performance demands of Data Center applications.

    The Mellanox ConnectX network adapter family deliver industry-leading connectivity for performance-driven server and storage applications.

    These ConnectX adapter cards enable high bandwidth coupled with ultra-low latency for diverse applications and systems, resulting in faster access and real-time responses.

    The Mellanox LinkX cables and transceivers product family provides the industry’s most complete line of 10, 25, 40, 50, 100, 200, and 400Gb interconnect products for Cloud, Web 2.0, Enterprise, telco, and storage data centers applications. They are often used to link TOR switches downwards to servers, storage & appliances and upwards in switch-to-switch applications.

     

    In the following design, we demonstrate how to achieve two TOR switches highly available network topology utilizing Mellanox SN2xxx Series switches.

     

     

    Solution Benefits

    The combined Mellanox ONYX - NEO solution provides:

    • Best ROI and faster time to market: with full automation, quickly deploy processes in your data center and provision new systems, the network infrastructure and applications.
      All this in a reliable manner, eliminating manual configuration errors.
    • Simplicity: NEO offers a rich set of predefined automation capabilities for network provision, including zero-touch provisioning and one-click provision.
      NEO offers automation for simple software upgrades across the Mellanox SN2000 series of switch systems and the Connect-X series of adapters.

    • Unified solution: NEO acts as a central configuration repository within the fabric by backing up all device configurations regularly.
    • Visibility: NEO provides in-depth visibility into network health, traffic utilization, and performance through various dashboards and heat maps consolidated in one pane of glass.

     

     

    Setup Overview

    In the setup we cover TOR switches - the Access network layer.

    Setup Design

    Terminology

    • Multichassis Link Aggregation (MLAG) is the ability of two and sometimes more switches to act like a single switch when forming link bundles. This allows a host to uplink to two switches for physical diversity, while still only having a single bundle interface to manage.

    • IPL (Inter Peer Link): This is the link between the two switches. The IPL link is required, and used for control and may be used for traffic in case of port failures. This link serves the most important role of transmitting keep alives between switches such that each switch knows that the other switch is still present. In addition, all mac-sync messages, IGMP groups sync and other DB sync messages are sent across this link. Hence it is critical to enable flow control on this link. Even if there is heavy congestion on this link, the control traffic will still get through.
    • Multi-Active Gateway Protocol (MAGP) designed to resolve the default gateway problem when a host is connected to a set of switch routers (SRs) via MLAG with no LACP control. The network functionality in that case requires that each SR is an active default gateway router to the host, thus reducing hops between the SRs and directly forwarding IP traffic to the L3 cloud regardless which SR traffic comes through.
    • What is the difference between MAGP and VRRP?
      MAGP is Mellanox proprietary protocol that implements active-active VRRP.

     

    Bill of Materials - BOM

    Physical Network Connections

    Please see below physical connection diagrams for each solutions.

    Solution 1 (up to 18 servers)

    Solution description

    A Mellanox SN2010 is used as TOR switch.

    Allow to scale up to 18 nodes in a rack and have total of 4 x 100GbE uplink ports to WAN/LAN connection.

    2 x 100GbE connection between TOR switches by using the QSFP28 100GbE Passive Copper Cables.

    Dedicated management port in each Mellanox switch connected to Switch Management Network.

    Single 25GbE connection from server to each TOR switch by using the SFP28 25GbE Passive Copper Cable.

    Solution 2 (up to 48 servers)

    Solution description

    A Mellanox SN2410 is used as TOR switch.

    Allow to scale up to 48 nodes in a rack and have total of 4 x 100GbE uplink ports to WAN/LAN connection.

    2 x 100GbE connection between TOR switches by using the QSFP28 100GbE Passive Copper Cables.

    Dedicated management port in each Mellanox switch connected to Switch Management Network.

    Single 25GbE connection from server to each TOR switch by using the SFP28 25GbE Passive Copper Cable.

     

    Ansible control machine must have an access to switch via management networks in order to provision, operate and orchestrate Ethernet fabric.

     

    NOTE: In this document we do not cover connectivity to corporate network.

    NOTE: We strongly recommend to use out-of-band management for Mellanox switches - use dedicated management port on each switch.

     

     

    Fabric Management Logical Diagram

    Please see below switch management logical diagram.

    We used NEO management software to provision, configure and monitor our network fabric.

     

    Dedicated management port in each Mellanox switch connected to Switch Management Network.

    Mellanox NEO must have an access to switch via management networks in order to provision, operate and orchestrate Ethernet fabric.

     

    NOTE: In this document we do not cover connectivity to corporate network.

    NOTE: We strongly recommend to use out-of-band management for Mellanox switches - use dedicated management port on each switch.

    Network Configuration Using Mellanox NEO

    NEO Virtual Appliance

    NEO software available for download as CentOS/RedHat installation package as well as Virtual Appliances for various virtualization platforms.

    NEO Virtual Appliance is available in various file formats compatible with leading virtualization platforms including VMware ESXi, Microsoft Hyper-V, Nutanix AHV, Red Hat Virtualization, IBM PowerKVM, and more.

     

    NEO Logical Schema

    Below provided logical connectivity schema between all Mellanox SW and HW components.

    MOFED and NEO-HOST is the optional Mellanox software components for host installation.

     

     

     

    Downloading Mellanox NEO

    Mellanox NEO is available for download from Mellanox NEO™ product page.

     

     

    You'll be asked to fulfill short form and download instructions will be sent to you email.

     

    Installing Virtual Appliance

    You are welcome to read the Mellanox NEO Quick Start Guide for detailed installation instructions.

    This Quick Start Guide provides step-by-step instructions for the Mellanox NEO™ software installation and Virtual Appliance deployment.

     

    NOTE: In our example we uses NEO Virtual Appliance that installed on VMware ESXi server.

     

    Once NEO VM is deployed you can connect to appliance console and use the following credentials(default) to login to your VM:

    • Username: root
    • Password: 123456

    After Login you can see appliance information screen like below.

    NOTE: The MAC address that is assigned to the VM must have DHCP record in order to get an IP address.

     

    Switch OS installation / configuration

    Please start from the HowTo Get Started with Mellanox switches guide if you are not familiar with Mellanox switch software.

    For more information please refer to the Mellanox Onyx User Manual located at support.mellanox.com or www.mellanox.com -> Switches

     

    NOTE: Before starting to use the Mellanox switches, we recommend that you upgrade the switches to the latest Mellanox Onyx™ version.

     

    You can download it from myMellanox - the Mellanox Support site. Please note, that you need active support subscription.

     

    Fabric configuration

    In this guide the Ethernet switch fabric is configured as Layer 2 Ethernet network.

    There are two ways to configure switches:

    • CLI based configuration done one-by-one on all switches
    • Wizard based configuration with Mellanox NEO

    If you aren't familiar with Mellanox NEO please refer to Mellanox NEO Solutions.

     

    Configuration Example for Solution-2 (up to 48 servers)

    Our example shows one rack configuration connectivity of the two SN2410 switches.

    In our example each TOR switch configured with 3 VLANs.

     

    Essential Configuration.

    Below described how to configure Ethernet Switch Fabric by using Mellanox NEO.

    1. Login to Mellanox NEO WEB UI using the following credentials(default):

    • Username: admin
    • Password: 123456

     

    NOTE: Mellanox NEO URL can be found in appliance console information screen.

     

    2. Register devices.

    Register all switches via "Add Devices" wizard in Managed Elements.

    1.JPG

     

    3. Configuring Mellanox Onyx Switch for LLDP Discovery

    Run Provisioning "Enable Link Layer Discovery..." from Task tab on all switches.

     

    4. Sure that discovery working properly. Click on " Network Map".

    2_2.JPG

     

    MTU Configuration.

    1. Setup NEO MTU Service

    The NEO MTU Service allows configuring a port MTU on specified Mellanox Onyx switches to a desirable value and periodically validates their configuration.

    • Click the “Add” button on the right side of the MTU row.
    • Fill in the name, description, and MTU fields. Click on "Next".

    • Choose the device to configure the MTU service for, and click “Finish”.

    • Once clicked “Finish”, a service instance will be created and a service element will appear on the Services main page.

    2. Configure MTU for network ports

    • A right click on a service element and click on "Apply Configuration".

     

    IPL Configuration.

    1. Configure a new MLAG Service (IPL configuration):

    • Click the “Add” button on the right side of the MLAG row.
    • In the Cluster tab, select the switch type and IP of the first switch in the cluster. The rest of the fields (including the collapsible Advanced section) will be filled out automatically, with the option to be edited.
      Note that some fields might not be filled in case there is no appropriate p
      eer switch.

    NOTE: The information in the Cluster tab is mandatory for the creation of the MLAG service, and cannot be changed once the service is created.

    3_1.JPG

     

    • Under Networks tab, you can manage MAGP networks on the MLAG cluster.
      Click “Add” to add a new network and fill in the required information, or edit/delete a network using the icons in the rightmost column of the network row.

    NOTE: Networks are not mandatory for the MLAG service creation. They can be added, edited or removed after the service has been created.

     

    3_2.JPG

    3_3.JPG

    3_4.JPG

    • Under Servers tab, you can't do nothing.

    NOTE: Servers are not mandatory for the MLAG service creation. They can be added, edited or removed after the service has been created.
    However, if you define a server, you also need to define the network it belongs to in the Networks tab.

    3_5.JPG

     

    2. IPL Configuration.

    • Right click on “MLAG_Service_1” button on the left side of the MLAG row and click on "Apply Configuration".

    • To Validate the configuration. Right click on “MLAG_Service_1” button on the left side of the MLAG row and click on "Validate".

     

    Check switch configuration

    Connect to switches by SSH and check running configuration:

    ssh admin@your_switch_ip

    Mellanox Onyx Switch Management

    Last login: Tue Nov 2 15:21:34 2010 from

     

    Mellanox Switch

     

    swx-vwd-11 [rack01-mlag-ipl-group: master] > ena

    swx-vwd-11 [rack01-mlag-ipl-group: master] # conf t

    swx-vwd-11 [rack01-mlag-ipl-group: master] (config) # show running-config

    ##

    ## Running database "initial"

    ## Generated at 2010/12/11 01:03:16 +0000

    ## Hostname: swx-vwd-11

    ##

     

     

    ##

    ## Running-config temporary prefix mode setting

    ##

    no cli default prefix-modes enable

     

     

    ##

    ## MLAG protocol

    ##

    protocol mlag

     

     

    ##

    ## Interface Ethernet configuration

    ##

    interface port-channel 1

    interface ethernet 1/1-1/54 mtu 9216 force

    interface ethernet 1/55-1/56 channel-group 1 mode active

     

     

    ##

    ## LAG configuration

    ##

    lacp

     

     

    ##

    ## VLAN configuration

    ##

    vlan 1611-1613

    vlan 4000

     

     

    ##

    ## STP configuration

    ##

    no spanning-tree

     

     

    ##

    ## L3 configuration

    ##

    ip routing vrf default

    interface vlan 1611

    interface vlan 1612

    interface vlan 1613

    interface vlan 4000

    interface vlan 1611 ip address 192.168.11.252/24 primary

    interface vlan 1612 ip address 192.168.12.252/24 primary

    interface vlan 1613 ip address 192.168.13.252/24 primary

    interface vlan 4000 ip address 10.10.10.2/24 primary

     

     

    ##

    ## DCBX PFC configuration

    ##

    dcb priority-flow-control enable force

    interface port-channel 1 dcb priority-flow-control mode on force

     

     

    ##

    ## LLDP configuration

    ##

    lldp

     

     

    ##

    ## MAGP configuration

    ##

    protocol magp

    interface vlan 1611 magp 82

    interface vlan 1612 magp 83

    interface vlan 1613 magp 84

    interface vlan 1611 magp 82 ip virtual-router address 192.168.11.254

    interface vlan 1612 magp 83 ip virtual-router address 192.168.12.254

    interface vlan 1613 magp 84 ip virtual-router address 192.168.13.254

    interface vlan 1611 magp 82 ip virtual-router mac-address 00:00:5E:00:01:01

    interface vlan 1612 magp 83 ip virtual-router mac-address 00:00:5E:00:01:02

    interface vlan 1613 magp 84 ip virtual-router mac-address 00:00:5E:00:01:03

     

     

    ##

    ## MLAG configurations

    ##

    mlag-vip neo-mlag-vip-4000 ip 192.168.1.1 /24 force

    no mlag shutdown

    mlag system-mac 00:00:5E:00:01:00

    interface port-channel 1 ipl 1

    interface vlan 4000 ipl 1 peer-address 10.10.10.1

     

     

    ##

    ## Logging configuration

    ##

    logging monitor events none

     

     

    ##

    ## AAA remote server configuration

    ##

    # ldap bind-password ********

    # radius-server key ********

    # tacacs-server key ********

     

     

    ##

    ## Network management configuration

    ##

    # web proxy auth basic password ********

     

     

    ##

    ## X.509 certificates configuration

    ##

    #

    # Certificate name system-self-signed, ID 14193208c7eab18b11b69771aea56e21dad9abc7

    # (public-cert config omitted since private-key config is hidden)

     

     

    ##

    ## Persistent prefix mode setting

    ##

    cli default prefix-modes enable

     

    Done!