L3 Network Design with OSPF at Scale with Mellanox NEO (Draft)

Version 24

    This post describes how-to configure Layer 3 Leaf-Spine topology with OSPF routing protocol by Mellanox NEO™.




    As new applications are constantly evolving, data centers must be flexible and future-proof to meet the demand for higher throughput and higher scalability. Traditionally, aggregation switching in data centers of Cloud providers, Web 2.0 providers, and large-scale enterprises has been based on modular switches.


    The Mellanox Virtual Modular Switch® solution (VMS) provides an ideal, optimized approach for aggregating racks. Each of the VMS switches runs Open Shortest Path First (OSPF) protocol or Border Gateway Protocol (BGP). To complement the OSPF/BGP routing decisions, the switches also run Equal-Cost Multi-Path routing (ECMP), which provides dynamic load balancing between routes of equal cost. By leveraging ECMP, congestion within the VMS and in the data center is avoided. Configuration of the VMS infrastructure is performed by the VMS Wizard via Mellanox NEO UI.



    Overview of Mellanox Components

    • The Mellanox Spectrum® switch family provides the most efficient network solution for the ever-increasing performance demands of Data Center applications.
    • The Mellanox ConnectX® network adapter family delivers industry-leading connectivity for performance-driven server and storage applications. These ConnectX adapter cards enable high bandwidth coupled with ultra-low latency for diverse applications and systems, resulting in faster access and real-time responses.
    • Mellanox NEO is a powerful platform for managing computing networks. It enables data center operators to efficiently provision, monitor, and operate the modern data center fabric.
    • Mellanox NEO-Host is a powerful solution for orchestration and management of host networking. NEO-Host is integrated with Mellanox NEO and can be deployed on Linux hosts managed by NEO.


    Related References



    Data Center Overview

    Data center networks have traditionally been built in a three-layer hierarchical tree consisting of access, aggregation, and core layers.




    Increasing east-west traffic within the data center (server-server, server-storage, etc.), an alternative to the traditional access-aggregation-core network model, is becoming more widely used. This architecture shown below, known as a Clos or leaf-spine network, is designed to minimize the number of hops between hosts.




    The aggregation and core layers are merged into the spine layer. Every leaf switch connects to every spine switch, ensuring that all leaf switches are only one hop away from one another in order to minimize latency and chances for bottlenecks in the network.



    Network Configuration


    Solution Physical Diagram

    In the solution physical diagram below, we used NEO management software to provision, configure, and monitor our network fabric.


    Solution Description

    • Leaf-spine topology: SN2700 as spine, and SN2410 as leaf switch
    • This allows scaling up to 672 nodes with a single-host link to switch: 14 racks, 48 servers per rack with 3:1 blocking ratio
    • 4x100GbE connection from leaf to spine switches, 2x100GbE per spine switch via QSFP28 100GbE passive copper cables
    • Dedicated management port in each Mellanox switch connected to a switch management network



    Mellanox NEO must have access to a switch via management networks in order to provision, operate, and orchestrate the Ethernet fabric.


    In this article we do not cover connectivity to corporate networks.


    We strongly recommend the use of out-of-band management for Mellanox switches. Use a dedicated management port on each switch.


    NEO Virtual Appliance

    NEO software is available for download as a CentOS/RedHat installation package as well as a Virtual Appliance for various virtualization platforms.

    NEO Virtual Appliance is available in various file formats compatible with leading virtualization platforms including VMware ESXi, Microsoft Hyper-V, Nutanix AHV, Red Hat Virtualization, IBM PowerKVM, and more.


    NEO Logical Schema

    A logical connectivity schema between all Mellanox software and hardware components is illustrated below.


    MOFED and NEO-HOST are optional Mellanox software components for host installation.



    Downloading Mellanox NEO

    Mellanox NEO is available for download from the Mellanox NEO™ product page.



    You'll be asked to fill a short form, after which download instructions are sent to your email.


    Installing Virtual Appliance

    You are welcome to read the Mellanox NEO Quick Start Guide for detailed installation instructions. This guide provides a step-by-step run down for the Mellanox NEO software installation and Virtual Appliance deployment.


    In our example we use NEO Virtual Appliance installed on Microsoft Hyper-V platform.


    Once NEO VM is deployed you can connect to the appliance console and use the following default credentials to log into your VM:

    • Username: root
    • Password: 123456


    After logging in, you'll see the appliance information screen:



    The MAC address that is assigned to the VM must have DHCP record in order to get an IP address.


    Switch OS Installation and Configuration

    Please start with the HowTo Get Started with Mellanox switches guide if you are not familiar with Mellanox switch software. For more information please refer to the Mellanox Onyx™ User Manual located at support.mellanox.com or www.mellanox.com -> Switches.


    Before starting to use the Mellanox switches, we recommend that you upgrade the switches to the latest Mellanox Onyx version and use the latest version of Mellanox NEO.


    You can download that from myMellanox - the Mellanox Support site. Please note, that you need active support subscription.


    Fabric Configuration

    In this solution the Ethernet switch fabric is configured as a Layer 3 Ethernet network.

    There are two ways to configure switches:

    • CLI based configuration performed on each individual switch
    • Wizard based configuration with Mellanox NEO

    If you aren't familiar with Mellanox NEO please refer to the Mellanox NEO Solutions article.


    Example Configuration

    Our example shows multi-rack configuration connectivity of two leaf switches to two spine switches. Each leaf switch is connected by 4 cables to the two spine switches (2 cables per spine switch).


    The following is a cross-switch port connectivity table of our example:


    Interface type

    Spine-1 switch

    Spine-2 switch

    Leaf-1 switch

    Leaf-2 switch


    Ports 1-2


    Ports 49-50


    Ports 1-2

    Ports 51-52

    OSPFPorts 3-4Ports 49-50
    Ports 3-4Ports 51-52


    The procedure below describes how to configure an Ethernet Switch Fabric with Mellanox NEO.


    1. Log into the Mellanox NEO Web UI using the following default credentials:

    • Username: admin
    • Password: 123456

    The Mellanox NEO URL can be found in the appliance console information screen.


    2. Register all switches via the "Add Devices" wizard in Managed Elements.



    3. Configuring Mellanox Onyx Switch for LLDP Discovery: Run Provisioning "Enable Link Layer Discovery..." from the Task tab on all switches.


    Please note that this step requires at least 15 minutes for ensure complete discovery.


    Once the discovery process is complete, you'd be able to see the fully discovered switches with interconnections in "Network Map".


    4. Creating OSPF Area with setup "L3 Network Provisioning":

    Add an "L3 Network Provisioning" service in the Virtual Modular Switch service section under the "Services" tab.

    Fill the required fields for complete wizard creation.

    • Provide a name and description of the service
    • Provide in the OSPF Subnet Reservation field, the subnet type used for allocating IP addresses to OSPF areas.



    After completing service creation, please right-click on a service element and click "Apply Configuration".


    5. After finishing the configuration, please review the port status on each switch. All configured and connected ports must light green.