High Availability using Mellanox Switches and Adapters

Version 20

    This post is an overview of different High Availability (HA) solutions using Mellanox Switches and Adapters.

     

    >>Learn how to configure MLAG for free on the Mellanox Academy

     

    References

     

    How can I achieve HA in my network?

    HA can be achieved in several levels, that can split to two major levels

    1. Service HA: duplicate software services on multiple servers, or run applications that can use multi-path.
    2. Hardware HA: Adding additional hardware adapters (interfaces) and switches.


    Service HA

    Run the HA in the application level. For example:

    • Run OpenStack controller application in several hosts in parallel.
    • Use Multipath for iSCSI/ISER block storage.


    Hardware HA

    HA via the hardware can be achieved in several sub levels:

     

    1. Adapter HA

    Use two interfaces on the host. it can be done in two ways:

    • Use one adapter card with two ports -> Interface HA
    • Usage two adapter cards with one port each -> Adapter HA

    From the logical port configuration (eth2, eth3 ...) there is no difference, from the performance point of view, two interfaces on the same PCI slot will not reach to 40GbEx2 (for example), but will be limited by the PCIe throughput. On most cases people will use one adapter card with two ports.

    2. Top Of Rack (ToR) HA

    ToR HA depends on the network topology used. Each server is connected with two interfaces to two ToR switches.

    The ToR switches can be configured as L2 switches or L3 switches.

     

    2.1 MLAG L2

    This is a classical solution for ToR HA when the network is L2. Each server is connected with two interfaces to two ToR switches.

    • The ToR switches are enabled with MLAG and connected to each other via IPL link.
    • The servers are configured with bond interface
    • The top switch at this point can be L2 or L3 switch with default gateway to external L3 network.
    • The configuration of the MLAG switches is similar towards the servers or the top switch. In other words, the top switch is configured with bond interface towards the MLAG switches, in a similar manner to the servers.

     

    mlag-l2.png

     

    To learn how to configure MLAG, refer to HowTo Configure MLAG on Mellanox Switches.

     

    2.2 MLAG + MAGP (L2 and L3)

    In case Layer 3 terminated on the ToR switches, redundancy can be achieved with MAGP. MAGP (active-active VRRP) is a trivial configuration in addition to the MLAG configuration. The MAGP is configured to have floating (virtual) IP address on both switches.

     

    mlag-l3.png

     

    To learn how to configure MLAG refer to HowTo Configure MLAG on Mellanox Switches

    To learn how to configure MAGP refer to HowTo Configure MAGP on Mellanox Switches

     

    2.3 MLAG + VRRP (L2 and L3)

    This is a possible configuration (similar topology as the MLAG with MAGP), but less recommended as VRRP is an active-standby protocol. In this case the standby switch does not forward data, unlike with MAGP implementation where both the switches are actively forwarding data.

     

    To learn how to configure MLAG refer to HowTo Configure MLAG on Mellanox Switches

    To learn how to configure VRRP refer to HowTo Configure VRRP on Mellanox Ethernet Switches

     

    2.4 MSTP L2

    MSTP is another option for ToR HA. The servers are configured with bonded interface towards the ToR switches.

    In such topology 2 VLANs (or more) are being used from each server to two ToRs enabled with MSTP.

    This setup, is good when the severs are connected to more than one network (VLANS). In addition, the MSTP root can be distributed on multiple switches based on the vlans assigned to the instance, such that there is load balancing of data traffic.

     

    mstp.png

     

     

    To learn how to configure MSTP, refer to L2 VMS Training Guide

     

    3. ToR to Spine HA

    In case Layer 2 domain extends to the Spine switch, HA can also be achieved at the Spine level, ensuring that all switches from ToR to Spine are fully redundant. If Layer 3 is implemented between spine and ToR, HA can be achieved with OSPF and ECMP.


    3.1 ToR MLAG (L2) to Spine MLAG+ MAGP (L2 and L3)

    This option assumes that the ToR switches are configured with MLAG (L2) while there are only two spines  that configure with MLAG+MAGP (L3).

    mlag-mlag-l3.png

     


    3.2 ToR MLAG+ MAGP (L2 and L3) to Spine OSPF (L3)

    This solution assumes MLAG + MAGP running on the ToR switches connected to L3 OSFP network (on the spines).

     

    ospf.png

     

    To learn how to configure MLAG refer to HowTo Configure MLAG on Mellanox Switches

    To learn how to configure MAGP refer to HowTo Configure MAGP on Mellanox Switches

    To learn how to configure OSFP refer to HowTo Configure OSPF on Mellanox Switches (Running-Config) and to the VMS Reference Guide located here.

     

    3.3 ToR MSTP (L2) to Spine MSTP + VRRP -> (L2 and L3)

    This solution adds VRRP to a basic MASP setup that described above. In this solution, the spine switches are configured with VRRP group per VLAN running in the network. Load balancing is  recommended to be configured between VRRP groups.

     

    mstp-vrrp.png

    To learn how to configure MSTP refer to L2 VMS Training Guide

    To learn how to configure VRRP refer to HowTo Configure VRRP on Mellanox Ethernet Switches