Mellanox Onyx BGP Deployment Guide

Version 6

    Data Center Overview

    Data center networks have traditionally been built in a three-layer hierarchical tree consisting of access, aggregation and core layers.

     

    Figure 1. Traditional three-layer hierarchy

     

    Increasing east-west traffic within the data center (server-server, server-storage, etc.), an alternative to the traditional access-aggregation-core network model is becoming more widely used. This architecture, shown below is known as a Clos or leaf-spine network and is designed to minimize the number of hops between hosts.

     

    Figure 2. Leaf-spine network

     

    The aggregation and core layers are merged into the spine layer. Every leaf switch connects to every spine switch, ensuring that all leaf switches are only one hop away from one another in order to minimize latency and chances for bottlenecks in the network.

     

    Leaf-spine Topology Overview

    The connections between leaf and spine switches can be layer 2 or layer 3. In both topologies, downstream connections to hosts are layer 2 and connections to external networks are layer 3.

    In this deployment guide we are going to focus on layer 3 with the following guidelines:

    • Each leaf switch connects to every spine switch in the topology
    • Hosts are always connected to leaf switches

    In some of the top-of-rack switches we have configured MLAG pairs for high availability in order to provide a wider range of deployment options.

    The bandwidth of the fabric may be increased by adding connections between leafs and spines as long as the spine layer has capacity for the additional connections.

     

    Layer 3 Leaf-Spine Topology

    In a layer 3 leaf-spine network, traffic between leafs and spines is routed. The layer 3 / layer 2 boundary is at the leaf switches. Spine switches are never connected to each other in a layer 3 topology. Equal cost multi-path routing (ECMP) is used to load balance traffic across the layer 3 network. Connections within racks from hosts to leaf switches are layer 2.

     

    Figure 3. Layer 3 leaf-spine topology

     

     

    Layer 3 Protocols Used in the Leaf-Spine Fabric

    In addition to the Border Gateway Protocol (BGP) described in detail in the following sections, the following protocols are deployed in the use cases described later in this document.

     

    BFD

    Bidirectional Forwarding Detection (BFD) is used to rapidly detect communication failures between two adjacent systems over a layer 3 link. It is a simple and lightweight replacement for existing routing protocol link state detection mechanisms. Though optional, use of BFD is considered a best practice for optimizing a leaf-spine network.  BFD provides forwarding path failure detection times on the order of milliseconds rather than seconds as with conventional routing protocols. It is independent of routing protocols and provides a consistent method of failure detection when used across a network. Networks converge faster because BFD triggers link state changes in the routing protocol sooner and more consistently.

    Note: BFD will be available on Mellanox Onyx from release 3.6.8000 in June 2018.

     

    ECMP

    In a leaf-spine topology, leaf switches are no more than one hop away from each other. As shown in the use cases below, leafs have two equal cost paths to each other leaf through each spine. ECMP is used for load balancing packets along these layer 3 multiple equal cost paths. ECMP is enabled on all leaf and spine switches, allowing traffic between leafs to be load balanced across the spines.

     

    Border Gateway Protocol (BGP) Overview

    Border Gateway Protocol (BGP) is a routing protocol which is designed to transfer routing information between routers, enabling the routing of data between any two endpoints in the network. It maintains and propagates a table of routes which designates network reachability among autonomous systems (ASs).  BGP neighbors, or peers, are routers configured manually to converse using the BGP protocol on top of a TCP session (port 179). A BGP speaker periodically sends keep-alive messages to maintain the connection. Network reachability includes such information as forwarding destinations (IPv4 or IPv6) together with a list of ASs that this information traverses and other attributes, so it becomes possible to construct a graph of AS connectivity without routing loops. BGP makes it possible to apply policy rules to enforce a connectivity graph.

     

    As indicated, layer 3 leaf-spine networks use ECMP for load balancing. eBGP and iBGP (described below) handle ECMP differently. By default, eBGP supports ECMP without any adjustments. iBGP requires a BGP route reflector and the use of the AddPath feature to fully support ECMP. To keep configuration complexity to a minimum, it is recommended to deploy eBGP in leaf-spine fabric deployments.

     

    BGP tracks IP reachability to the peer remote address and the peer local address. Whenever either address becomes unreachable, BGP brings down the session with the peer. To ensure fast convergence with BGP, it is recommended to enable fast fail-over with BGP. Fast fail-over terminates external BGP sessions of any directly adjacent peer if the link to reach the peer goes down without waiting for the hold-down timer to expire.

     

    iBGP and eBGP

    When BGP runs between two peers in the same AS, it is referred to as Internal BGP (iBGP, or Interior Border Gateway Protocol). When BGP runs between separate ASs, it is called External BGP (eBGP, or Exterior Border Gateway Protocol). Both sides can initiate a connection, after the initial connectivity is created, and the BGP state machine drives both sides to enter into the ESTABLISHED state where they can exchange UPDATE messages with reachability information.

     

    In the sections below we will describe some of the major components and commands essential for the deployment and maintenance of BGP in a leaf spine topology employing Mellanox switches.

     

    BGP Route-Reflectors

    Because of the iBGP full-mesh requirement, most networks use route reflectors to simplify configuration and allow iBGP to scale in large networks. By this configuration, all peers should establish neighborship only with their RR instead of with all iBGP peers. RR allows an iBGP router to advertise (reflect) the best routes it learned from one iBGP neighbor to another.

     

    All iBGP peers that connect to RR are classified as Route-Reflector clients. Every other iBGP router that is not a RR or a RR client is classified as a non-client peer. An RR along with its client peers form a cluster. Each cluster can have multiple RRs which help avoid a single point of failure and achieve redundancy. It is also possible to have multiple RRs within an AS where each RR is a non-client peer to another RR.

     

    Route reflectors advertise received routes as the following:

    • From eBGP neighbors: The best routes are propagated to all BGP neighbors including other RRs, client peers, and non-client peers.
    • From non-client peers: The best routes are reflected to all the client peers as well as to the eBGP peers.
    • From client peers: The best routes are reflected to all non-client peers as well as to the client peers.

     

    Below is a route reflector configuration example (please refer to the leaf-spine topology use case below).

    Note: This document provides Mellanox Onyx configuration examples based on the deployment use cases presented later in this document.

    CLI configuration of route-reflector-clients

    Spine-2 [standalone: master] (config) # router bgp 65021 neighbor iBGP peer-group

    Spine-2 [standalone: master] (config) # router bgp 65021 vrf default neighbor 10.0.0.11 peer-group iBGP

    Spine-2 [standalone: master] (config) # router bgp 65021 vrf default neighbor 10.0.0.12 peer-group iBGP

    Spine-2 [standalone: master] (config) # router bgp 65021 vrf default neighbor 10.0.0.13 peer-group iBGP

    Spine-2 [standalone: master] (config) # router bgp 65021 neighbor iBGP route-reflector-client

     

    BGP Maximum-Paths

    In order to perform ECMP in BGP (eBGP/iBGP), you may need to define how many equal cost routes BGP will install into the routing table. Below is a maximum-path configuration example (please refer to the leaf-spine topology use case below).

    CLI configuration of maximum-paths for iBGP

    Leaf-3 [standalone: master] (config) # router bgp 65011 vrf default maximum-paths ibgp 2

    CLI to configure maximum-paths for eBGP

    Leaf-3 [standalone: master] (config) # router bgp 65011 vrf default maximum-paths 2

     

    BGP Multipath Relax

    In order to perform ECMP in BGP with different as-path values (same length though), you may need to configure as-path multipath-relax (if you're using eBGP). This command will ignore as-path values of the as-path attribute in BGP best route selection and will allow installing routes with different as-paths into the routing table assuming they are equal (if they are also equal with all attributes until the as-path attribute).

    Below is a multipath relax configuration example (please refer to the leaf-spine topology configuration below).

    CLI to configure multipath Relax

    Leaf-3 [standalone: master] (config) # router bgp 65021 vrf default bestpath as-path multipath-relax force

     

    BGP Next-Hop-Self

    The next-hop attribute defines the next-hop router IP address that should be used to reach the destinations of received BGP routes. When a route was received from outside the AS (eBGP) and forwarded into the AS (iBGP), the next-hop of the route will not change. This is because the iBGP router does not change the next-hop when forwarding prefixes received from a eBGP neighbor into the AS. This can cause connectivity problems since in most of the time an iBGP router inside the AS does not have a route to an external eBGP router (the next hop of the received route). In order to force an iBGP router change those next-hop prefixes, you need to configure this option towards the neighbors you need (internal iBGP routers). We have not implemented such a scenario in our leaf/spine usage example since it uses route-reflectors (that change next-hop automatically) and lacks eBGP routers.

    CLI configuration of next-hop-self towards neighbor/peer-group

    Switch [standalone: master] (config) # router bgp 65021 vrf default neighbor 1.1.1.1 peer-group iBGP

    Switch [standalone: master] (config) # router bgp 65021 vrf default neighbor iBGP next-hop-self

    Switch [standalone: master] (config) # router bgp 65021 vrf default neighbor 1.1.1.1 remote-as 65021

    Switch [standalone: master] (config) # router bgp 65021 vrf default neighbor 1.1.1.1 next-hop-self

     

    BGP Peer Group

    Often in a BGP protocol (especially iBGP), many neighbors are configured with the same update policies (e.g. same route maps, distribute lists, filter lists, update source, and so on). Neighbors with the same update policies can be grouped into peer-groups to simplify configuration and make update calculation more efficient. In order to create peer-groups, you may need to configure neighbors and add them into the created peer-groups. Below is a peer group configuration example (please refer to the leaf-spine topology configuration below).

    CLI configuration of peer-groups

    Spine-2 [standalone: master] (config) # router bgp 65021 neighbor iBGP peer-group

    Spine-2 [standalone: master] (config) # router bgp 65021 vrf default neighbor 10.0.0.11 remote-as 65021

    Spine-2 [standalone: master] (config) # router bgp 65021 vrf default neighbor 10.0.0.11 peer-group iBGP

     

    Example of same update policies configuration for peer-groups:

    Spine-2 [standalone: master] (config) # router bgp 65021 neighbor iBGP update-source loopback 0

    Spine-2 [standalone: master] (config) # router bgp 65021 neighbor iBGP route-reflector-client

     

    BGP Neighbors Specified with IP Range

    BGP dynamic neighbors is a way to establish BGP neighbors without specifically defining the neighbors' remote IP addresses. Using the BGP listen range command you specify a range of IP addresses of devices that you trust to become BGP neighbors with you. When a TCP request on port 179 is received from one of those trusted IPs, a new BGP neighbor is dynamically created.

    In order to establish BGP neighborship using dynamic neighbors, you will need to create a dynamic peer-group and set an IP listen range to it. Below is a BGP listen range configuration example (please refer to the leaf-spine topology configuration below).

    CLI configuration of BGP listen range (must be configured on a peer-group)

    *Let's take Spine-1 in iBGP topology for example. We can replace the existing peer-group with static neighbors with a dynamic peer-group that has a listen range of 10.0.0.0 /24

    Before:

    router bgp 65021 neighbor iBGP peer-group

    router bgp 65021 vrf default neighbor 10.0.0.11 remote-as 65021

    router bgp 65021 vrf default neighbor 10.0.0.12 remote-as 65021

    router bgp 65021 vrf default neighbor 10.0.0.13 remote-as 65021

    router bgp 65021 vrf default neighbor 10.0.0.11 peer-group iBGP

    router bgp 65021 vrf default neighbor 10.0.0.12 peer-group iBGP

    router bgp 65021 vrf default neighbor 10.0.0.13 peer-group iBGP

     

    After:

    Spine-1 [standalone: master] (config) # router bgp 65021 vrf default neighbor Listen-group peer-group

    Spine-1 [standalone: master] (config) # router bgp 65021 vrf default bgp listen range 10.0.0.0 /24 peer-group Listen-group remote-as 65021

    *dynamic peer group has the same options as static peer-group:

    e.g.

    Spine-1 [standalone: master] (config) # router bgp 65021 neighbor Listen-group update-source loopback 0

    Spine-1 [standalone: master] (config) # router bgp 65021 neighbor Listen-group route-reflector-client

     

    Route-maps with BGP

    Route-map in BGP allows you to filter and/or change incoming/outgoing routing updates from or to neighbors. Route-map makes the filtering/changes in BGP routing updates by matching a parameter(s) in the update (e.g. IP address, as-path, metric, etc.) and setting different or additional parameters to the routes (e.g. local-pref, weight, metric, etc.).

     

    Route-map can permit or deny the routes (like ACLs). Routes that have a "match" in the "permit" route-map statement will get the "set" options (if configured), otherwise they will continue to the next route-map statement (deny route-map will do the same but only for routes that don't match to any condition configured under "match" action). As mentioned before, for filtering updates you need to use only the "match" option.

     

    Route-map uses additional filtering tools to match parameters, i.e. ip prefix-lists and ip community-lists.

    In order to filter/change prefixes sent or received from a neighbor, route-map must be attached to a neighbor statement under the bgp configuration.

    Below is  a filter routes by route-maps example (please refer to the leaf-spine topology configuration below).

    CLI to filter routes by route-maps

    e.g. Leaf-3 wants to filter outgoing LAN HOST route to Spine-1

    We will use a prefix-list to match HOST IP address.

    Leaf-3 [standalone: master] (config) # ip prefix-list HOST101 permit 172.16.2.101 /32

    Leaf-3 [standalone: master] (config) # route-map FILTER_HOST101 deny 1 match ip address HOST101

    Leaf-3 [standalone: master] (config) # router bgp 65021 neighbor 10.0.0.21 route-map FILTER_LAN out

     

    **Route-map can be attached to a neighbor for inbound/outbound and both directions.

     

    In case we want to change a few parameters in the route before it advertised to Spine-1 (e.g. metric, next-hop and local-pref):

    Leaf-3 [standalone: master] (config) # ip prefix-list HOST101 permit 172.16.2.101 /32

    Leaf-3 [standalone: master] (config) # route-map MODIFY_HOST101 permit 1 match ip address HOST101

    Leaf-3 [standalone: master] (config) # route-map MODIFY_HOST101 permit 1 set metric 123

    Leaf-3 [standalone: master] (config) # route-map MODIFY_HOST101 permit 1 set ip next-hop 1.2.3.4

    Leaf-3 [standalone: master] (config) # route-map MODIFY_HOST101 permit 1 set local-preference 200

    Leaf-3 [standalone: master] (config) # router bgp 65021 neighbor 10.0.0.21 route-map MODIFY_HOST101 out

     

    Redistributing Routes in BGP

    Redistribution allows to advertise routes into BGP, a route that was learned by other means, such as another routing protocol (e.g. OSPF), static routes or directly connected ones.

    In order to redistribute routes into BGP, you need to decide which routes will be advertised into BGP.

    CLI to redistribute connected* networks

    *Redistribution into BGP can be don for [connected/ospf/ospf-external/ ospf-internal/static]

    Leaf-3 [standalone: master] (config) # router bgp 65021 vrf default redistribute connected

     

    BGP BFD

    Since BGP is a stable but slow routing protocol, usage of BFD protocol over it significantly decreases neighbors' failover time. BFD establishes an additional session between the neighbors with shorter timers (ms) to detect failure faster. For using BFD over BGP you need to configure BFD on the neighbors/peer-groups. BFD configuration must be on both sides to create the session.

    Below is a BFD over BGP configuration example (please refer to the leaf-spine topology configuration below).

    Note: BFD will be available on Mellanox Onyx from release 3.6.8000 in June 2018.

    CLI configuration of BFD over BGP neighbors/peer-groups

    Spine-1 [standalone: master] (config) # router bgp 65021 vrf default neighbor iBGP fall-over bfd

    Leaf-2 [standalone: master] (config) # router bgp 65011 vrf default neighbor 10.0.0.21 fall-over bfd

     

    Troubleshooting

    BGP Neighbor States, Timers, Route Advertisements and Prefix Counters

    Common commands may be used  to troubleshoot BGP neighbor states, as-numbers and the amount of sent/received routes. These commands include "show ip bgp summary" and "show ip bgp neighbors [neighbor IP]" as shown below.

    Spine-1 [standalone: master] # show ip bgp summary

    BGP router identifier 10.0.0.22, local AS number 65021

    BGP table version is 77653, main routing table version 77653

    --------  -  -- ------- -------  ------  --- ---- -------     ------------

    Neighbor  V  AS MsgRcvd MsgSent  TblVer  InQ OutQ Up/Down     State/PfxRcd

    --------  -  -- ------- -------  ------  --- ---- -------     ------------

    10.0.0.11 4 65021 48874  48674   77653    0    0   2:21:37:07  CONNECT

    10.0.0.12 4 65021 32730  60964   77653    0    0   3:23:27:23  IDLE

    10.0.0.13 4 65021 32730  60964   77653    0    0   3:23:27:23  ESTABLISHED

     

    Spine-1 [standalone: master] # show ip bgp neighbors 10.0.0.13

    BGP neighbor: 10.0.0.13, remote AS: 65021, link: internal                 

       BGP version: 4, remote router ID: 10.0.0.13                               

       BGP State: ESTABLISHED                                                 

       Last read: 0:22:25:00, last write: 0:22:25:11, hold time is: 180, keepalive interval in seconds: 60

       Configured hold time in seconds: 180, keepalive interval in seconds: 60                          

       Minimum holdtime from neighbor in seconds: 180                                                   

     

       Neighbor capabilities:

         Route refresh: advertise and received

         Graceful Restart Capability: advertise and received

         Address family IPv4 Unicast: advertise and received

         Address family IPv6 Unicast: n/a

     

       Message statistics:

         InQ depth is: 0

         OutQ depth is: 0

                              ----       ------

                              Sent       Rcvd

                              ----       ------

         Opens:                  1          1

         Notifications:          0          0

         Updates:            56230      28012

         Keepalives:          4740       4724

         Route Refresh:          0          0

         Total:              60971      32737

       Default minimum time between advertisement runs in seconds: 30

     

       For address family: IPv4 Unicast

       BGP table version: 77653

       Output queue size : 0

                                      ----       ----

                                      Sent       Rcvd

       Prefix activity                ----       ----

         Prefixes Current:              73         70

         Prefixes Total:                73      11600

         Implicit Withdraw:          36524       1400

         Explicit Withdraw:              0      11530

         Used as bestpath:             n/a         70

         Used as multipath:            n/a        n/a

     

                                        --------    -------

                                        Outbound    Inbound

       Local Policy Denied Prefixes     --------    -------

         Total:                                0          0

     

       Connections established: 1; dropped: 2

       Last reset: 3:23:33:27, due to: 0 (0)

    Maximum hops to external BGP neighbor: 1

    Connection state is: ESTABLISHED

    Local host: 10.0.0.22, Local port: 179

    Foreign host: 10.0.0.13, Foreign port: 55023

     

    BGP Table

    For troubleshooting BGP advertisements (local and received prefixes), their as-numbers and other route-attributes it is recommended to use the "show ip bgp" command. This show command presents the BGP table including all prefixes received/advertised by BGP. It is also possible to see which route is chosen as the best route, origin of the route (eBGP/iBGP), routes' next-hop, etc.

    Spine-1 [standalone: master] # show ip bgp

    BGP table version is: 77653, local router ID is: 10.0.0.22

    Status codes: s suppressed, d damped, h history,* valid,> best, i - internal

    r RIB-failure, S Stale, m multipath, b backup-path, x best-external

    Origin codes: i - IGP, e - EGP, ? - incomplete                                

     

        -------          --------           ------     ------   ------ ----

        Network          Next Hop           Metric     LocPrf   Weight Path

        -------          --------           ------     ------   ------ ----

    *>  1.2.3.4/32      10.0.0.13              10        100        0  65021 ?

    *>  6.6.6.6/32      10.0.0.13              11        100        0  65021 ?

    *>  7.7.7.7/32      10.0.0.13              11        100        0  65021 ?

    *>  10.0.0.22/32      0.0.0.0               0        100    32768        i

     

    BGP-related RFCs Supported on Mellanox Onyx

     

    DescriptionRFC
    BGP Communities AttributeRFC1997
    Protection of BGP Sessions via the TCP MD5 Signature OptionRFC2385, requires MD5 capable IP stack
    Use of BGP-4 Multiprotocol Extensions for IPv6 Inter-Domain RoutingRFC2545
    Route Refresh Capability for BGP-4RFC2918
    BGP-4RFC4271
    BGP Extended Communities AttributeRFC4360
    BGP Route Reflection: An Alternative to Full Mesh Internal BGP (IBGP)RFC4456
    Subcodes for BGP Cease Notification MessageRFC4486
    Graceful Restart Mechanism for BGPRFC4724
    Multiprotocol Extensions for BGP-4RFC4760
    BGP Support for Four-octet AS Number SpaceRFC4893
    Textual Representation of Autonomous System (AS) NumbersRFC5396
    Capabilities Advertisement with BGP-4RFC5492

     

     

    BGP Deployment Use Cases

    This section provides you with detailed configuration for two common BGP deployments in Mellanox switches running Mellanox Onyx.

    The first topology is deployed with iBGP, having all the switches sharing the same AS. In this iBGP deployment use case, the spine switches are used as active/backup route reflectors in order to facilitate the full mesh connectivity required in iBGP. The high availability between the two switches (leaf-1 7 leaf-2 to which Host-1 is connected) is achieved by MLAG.

     

    In the second topology, eBGP is used and therefore a different AS is assigned per switch. Note that in order to simplify the configuration, eliminate the path hunting and provide similar configurations per switch level in the rack, the following guidelines can be followed when eBGP is deployed:

    • AS numbering
      • 4-byte as-numbers (ASNs) should be used
      • All ToR routers are assigned their own ASN
      • Leafs across a pod have a different ASN, but leafs within each pod have an ASN that is unique to that pod
      • Spines share a common ASN
    • Adjust timers to achieve fast convergence
      • Advertisement-interval is 0

     

    iBGP Leaf Spine Deployment with MLAG (using Route Reflector)

    Provided below are the configuration settings for the switches shown in the figure below.

    Figure 4. iBGP deployment example

     

    Spine-1 configuration

    ## L3 configuration                                                                                        

    ##                                                                                                         

       ip routing vrf default                                                                                  

       interface ethernet 1/5 no switchport force                                                              

       interface ethernet 1/8 no switchport force                                                              

       interface ethernet 1/9 no switchport force                                                              

       interface loopback 0                                                                                    

       interface ethernet 1/5 ip address 100.0.0.2 255.255.255.0                                               

       interface ethernet 1/8 ip address 100.30.0.2 255.255.255.0                                              

       interface ethernet 1/9 ip address 100.20.0.2 255.255.255.0                                              

       interface loopback 0 ip address 10.0.0.21 255.255.255.255                                              

                                                                                                               

    ##                                                                                                         

    ## OSPF configuration                                                                                      

    ##                                                                                                         

       protocol ospf                                                                                           

       router ospf 1 vrf default                                                                               

       router ospf 1 vrf default router-id 10.0.0.21                                                           

       interface ethernet 1/5 ip ospf area 0.0.0.0                                                             

       interface ethernet 1/8 ip ospf area 0.0.0.0                                                             

       interface ethernet 1/9 ip ospf area 0.0.0.0

       interface loopback 0 ip ospf area 0.0.0.0

     

    ##

    ## BGP configuration

    ##

       protocol bgp

       router bgp 65021 vrf default

       router bgp 65021 vrf default router-id 10.0.0.21 force

       router bgp 65021 neighbor iBGP peer-group

       router bgp 65021 vrf default neighbor 10.0.0.11 remote-as 65021

       router bgp 65021 vrf default neighbor 10.0.0.12 remote-as 65021

       router bgp 65021 vrf default neighbor 10.0.0.13 remote-as 65021

       router bgp 65021 vrf default neighbor 10.0.0.11 peer-group iBGP

       router bgp 65021 vrf default neighbor 10.0.0.12 peer-group iBGP

       router bgp 65021 vrf default neighbor 10.0.0.13 peer-group iBGP

            router bgp 65021 neighbor iBGP update-source loopback 0

       router bgp 65021 neighbor iBGP route-reflector-client

     

    Spine-2 configuration

    ## L3 configuration                                                                    

    ##                                                                                     

       ip routing vrf default                                                              

       interface ethernet 1/13 no switchport force                                         

       interface ethernet 1/15 no switchport force                                         

       interface ethernet 1/25 no switchport force                                         

       interface loopback 0                                                                

       interface ethernet 1/13 ip address 100.40.0.2 255.255.255.0                         

       interface ethernet 1/15 ip address 100.10.0.2 255.255.255.0                         

       interface ethernet 1/25 ip address 100.50.0.2 255.255.255.0                         

       interface loopback 0 ip address 10.0.0.22 255.255.255.255                           

                                                                                           

    ##                                                                                     

    ## OSPF configuration                                                                  

    ##                                                                                     

       protocol ospf                                                                       

       router ospf 1 vrf default                                                           

       router ospf 1 vrf default router-id 10.0.0.22                                       

       interface ethernet 1/13 ip ospf area 0.0.0.0                                        

       interface ethernet 1/15 ip ospf area 0.0.0.0                                        

       interface ethernet 1/25 ip ospf area 0.0.0.0

       interface loopback 0 ip ospf area 0.0.0.0

     

    ##

    ## BGP configuration

    ##

       protocol bgp

       router bgp 65021 vrf default

       router bgp 65022 vrf default router-id 10.0.0.22 force

       router bgp 65021 neighbor iBGP peer-group

       router bgp 65021 vrf default neighbor 10.0.0.11 remote-as 65021

       router bgp 65021 vrf default neighbor 10.0.0.12 remote-as 65021

       router bgp 65021 vrf default neighbor 10.0.0.13 remote-as 65021

       router bgp 65021 vrf default neighbor 10.0.0.11 peer-group iBGP

       router bgp 65021 vrf default neighbor 10.0.0.12 peer-group iBGP

       router bgp 65021 vrf default neighbor 10.0.0.13 peer-group iBGP

            router bgp 65021 neighbor iBGP update-source loopback 0

       router bgp 65021 neighbor iBGP route-reflector-client

     

    Leaf-1 configuration

    ## MLAG protocol

    ##          

       protocol mlag

                

    ##          

    ## Interface Ethernet configuration

    ##                             

       interface mlag-port-channel 1

       interface port-channel 4000 

       interface ethernet 1/17 mlag-channel-group 1 mode active

       interface ethernet 1/25-1/26 channel-group 4000 mode active                            

       interface mlag-port-channel 1 no shutdown              

                                                              

    ##                                                        

    ## LAG configuration                                      

    ##                                                        

       lacp                                                   

       interface ethernet 1/17 lacp rate fast                 

                                                              

    ##                                                        

    ## VLAN configuration                                     

    ##                                                        

       vlan 172                                               

       vlan 4000                                              

       interface mlag-port-channel 1 switchport access vlan 172

                                                              

    ##                                                        

    ## STP configuration                                      

    ##                                                        

    no spanning-tree                                          

                                                              

    ##                                                        

    ## L3 configuration                                       

    ##                                                        

       ip routing vrf default                                 

       interface ethernet 1/9 no switchport force             

       interface ethernet 1/13 no switchport force            

       interface loopback 0                                   

       interface vlan 172                                     

       interface vlan 4000                                    

       interface ethernet 1/9 ip address 100.20.0.1 255.255.255.0

       interface ethernet 1/13 ip address 100.40.0.1 255.255.255.0

       interface loopback 0 ip address 10.0.0.12 255.255.255.255

       interface vlan 172 ip address 172.16.1.253 255.255.255.0

       interface vlan 4000 ip address 1.1.1.2 255.255.255.252 

                                                              

    ##                                                        

    ## OSPF configuration                                     

    ##                                                        

       protocol ospf                                          

       router ospf 1 vrf default                              

       router ospf 1 vrf default router-id 10.0.0.12          

       interface ethernet 1/9 ip ospf area 0.0.0.0            

       interface ethernet 1/13 ip ospf area 0.0.0.0           

       interface loopback 0 ip ospf area 0.0.0.0              

                                                              

    ##                                                        

    ## MAGP configuration                                     

    ##                                                        

       protocol magp                                          

       interface vlan 172 magp 172                            

       interface vlan 172 magp 172 ip virtual-router address 172.16.1.254

       interface vlan 172 magp 172 ip virtual-router mac-address AA:BB:CC:DD:EE:FF

                                                                              

    ##                                                                        

    ## MLAG configurations                                                    

    ##                                                                        

       mlag-vip MLAGCLUSTER ip 123.123.123.123 /30 force                      

    no mlag shutdown                                                          

       interface port-channel 4000 ipl 1                                      

       interface vlan 4000 ipl 1 peer-address 1.1.1.1                         

                                                                              

    ##                                                                        

    ## BGP configuration                                                      

    ##                                                                        

       protocol bgp

       router bgp 65021 vrf default

       router bgp 65021 vrf default router-id 10.0.0.12 force

       router bgp 65021 vrf default maximum-paths ibgp 2

       router bgp 65021 vrf default neighbor 10.0.0.21 remote-as 65021

       router bgp 65021 vrf default neighbor 10.0.0.22 remote-as 65021

       router bgp 65021 vrf default neighbor 10.0.0.21 update-source loopback 0

       router bgp 65021 vrf default neighbor 10.0.0.22 update-source loopback 0

       router bgp 65012 vrf default redistribute connected

     

    Leaf-2 configuration

    ## MLAG protocol                      

    ##                                    

       protocol mlag                      

                                          

    ##                                    

    ## Interface Ethernet configuration   

    ##                                    

       interface mlag-port-channel 1      

       interface port-channel 4000        

       interface ethernet 1/1 mlag-channel-group 1 mode active

       interface ethernet 1/13-1/14 channel-group 4000 mode active           

       interface mlag-port-channel 1 no shutdown              

                                                              

    ##                                                        

    ## LAG configuration                                      

    ##                                                        

       lacp                                                   

       interface ethernet 1/1 lacp rate fast                  

                                                              

    ##                                                        

    ## VLAN configuration                                     

    ##                                                        

       vlan 172                                               

       vlan 4000                                              

       interface mlag-port-channel 1 switchport access vlan 172

                                                              

    ##                                                        

    ## STP configuration                                      

    ##                                                        

    no spanning-tree                                          

                                                              

    ##                                                        

    ## L3 configuration                                       

    ##                                                        

       ip routing vrf default                                 

       interface ethernet 1/5 no switchport force             

       interface ethernet 1/8 no switchport force             

       interface loopback 0                                   

       interface vlan 172                                     

       interface vlan 4000                                    

       interface ethernet 1/5 ip address 100.50.0.1 255.255.255.0

       interface ethernet 1/8 ip address 100.30.0.1 255.255.255.0

       interface loopback 0 ip address 10.0.0.11 255.255.255.255

       interface vlan 172 ip address 172.16.1.252 255.255.255.0

       interface vlan 4000 ip address 1.1.1.1 255.255.255.252 

                                                                                                                                                            

    ##                                                        

    ## OSPF configuration                                     

    ##                                                        

       protocol ospf                                          

       router ospf 1 vrf default                              

       router ospf 1 vrf default router-id 10.0.0.11          

       interface ethernet 1/5 ip ospf area 0.0.0.0            

       interface ethernet 1/8 ip ospf area 0.0.0.0            

       interface loopback 0 ip ospf area 0.0.0.0              

                                                              

    ##                                                        

    ## MAGP configuration                                     

    ##                                                        

       protocol magp                                          

       interface vlan 172 magp 172                            

       interface vlan 172 magp 172 ip virtual-router address 172.16.1.254

       interface vlan 172 magp 172 ip virtual-router mac-address AA:BB:CC:DD:EE:FF

                                                                              

    ##                                                                        

    ## MLAG configurations                                                    

    ##                                                                        

       mlag-vip MLAGCLUSTER ip 123.123.123.123 /30 force                      

    no mlag shutdown                                                          

       interface port-channel 4000 ipl 1                                      

       interface vlan 4000 ipl 1 peer-address 1.1.1.2                         

                                                                              

    ##                                                                        

    ## BGP configuration                                                      

    ##                                                                        

       protocol bgp

       router bgp 65021 vrf default

       router bgp 65021 vrf default router-id 10.0.0.11 force

       router bgp 65021 vrf default maximum-paths ibgp 2

       router bgp 65021 vrf default neighbor 10.0.0.21 remote-as 65021

       router bgp 65021 vrf default neighbor 10.0.0.22 remote-as 65021

       router bgp 65021 vrf default neighbor 10.0.0.21 update-source loopback 0

       router bgp 65021 vrf default neighbor 10.0.0.22 update-source loopback 0

       router bgp 65021 vrf default redistribute connected

     

    Leaf-3 configuration

    ## L3 configuration                     

    ##                                      

       ip routing vrf default               

       interface ethernet 1/5 no switchport force

       interface ethernet 1/25 no switchport force

       interface ethernet 1/49 no switchport force

       interface loopback 0                   

       interface ethernet 1/5 ip address 100.0.0.1 255.255.255.0

       interface ethernet 1/25 ip address 100.10.0.1 255.255.255.0

       interface ethernet 1/49 ip address 172.16.2.254 255.255.255.0

       interface loopback 0 ip address 10.0.0.13 255.255.255.255

                                                                

    ##                                                          

    ## OSPF configuration                                       

    ##                                                          

       protocol ospf                                            

       router ospf 1 vrf default                                

       router ospf 1 vrf default router-id 10.0.0.13            

       interface ethernet 1/5 ip ospf area 0.0.0.0              

       interface ethernet 1/25 ip ospf area 0.0.0.0             

       interface loopback 0 ip ospf area 0.0.0.0

     

    ##

    ## BGP configuration

    ##

       protocol bgp

       router bgp 65021 vrf default

       router bgp 65021 vrf default router-id 10.0.0.13 force

       router bgp 65021 vrf default maximum-paths ibgp 2

       router bgp 65021 vrf default neighbor 10.0.0.21 remote-as 65021

       router bgp 65021 vrf default neighbor 10.0.0.22 remote-as 65021

       router bgp 65021 vrf default neighbor 10.0.0.21 update-source loopback 0

       router bgp 65021 vrf default neighbor 10.0.0.22 update-source loopback 0

       router bgp 65021 vrf default redistribute connected

     

    eBGP Leaf Spine Deployment with MLAG

    Provided below are the configuration settings for the switches shown in the figure below.

     

    Figure 5. eBGP deployment example

     

    Provided below are the configuration examples for each of the switches in the eBGP leaf-spine deployment example.

     

    Spine-1 configuration

    ## L3 configuration                                                                                  

    ##                                                                                                   

       ip routing vrf default                                                                            

       interface ethernet 1/5 no switchport force                                                        

       interface ethernet 1/8 no switchport force                                                        

       interface ethernet 1/9 no switchport force                                                        

       interface loopback 0                                                                              

       interface ethernet 1/5 ip address 100.0.0.2 255.255.255.0                                         

       interface ethernet 1/8 ip address 100.30.0.2 255.255.255.0                                        

       interface ethernet 1/9 ip address 100.20.0.2 255.255.255.0                                        

       interface loopback 0 ip address 10.0.0.21 255.255.255.255                                         

                                                                                                         

    ##                                                                                                   

    ## OSPF configuration                                                                                

    ##                                                                                                   

       protocol ospf                                                                                     

       router ospf 1 vrf default                                                                         

       router ospf 1 vrf default router-id 10.0.0.21                                                     

       interface ethernet 1/5 ip ospf area 0.0.0.0                                                       

       interface ethernet 1/8 ip ospf area 0.0.0.0                                                       

       interface ethernet 1/9 ip ospf area 0.0.0.0

       interface loopback 0 ip ospf area 0.0.0.0

     

    ##

    ## BGP configuration

    ##

       protocol bgp

       router bgp 65021 vrf default

       router bgp 65021 vrf default router-id 10.0.0.21 force

       router bgp 65021 vrf default neighbor 10.0.0.11 remote-as 65011

       router bgp 65021 vrf default neighbor 10.0.0.12 remote-as 65012

       router bgp 65021 vrf default neighbor 10.0.0.13 remote-as 65013

       router bgp 65021 vrf default neighbor 10.0.0.11 update-source loopback 0

       router bgp 65021 vrf default neighbor 10.0.0.12 update-source loopback 0

       router bgp 65021 vrf default neighbor 10.0.0.13 update-source loopback 0

     

    Spine-2 configuration

    ## L3 configuration                                                              

    ##                                                                               

       ip routing vrf default                                                        

       interface ethernet 1/13 no switchport force                                   

       interface ethernet 1/15 no switchport force                                   

       interface ethernet 1/25 no switchport force                                   

       interface loopback 0                                                          

       interface ethernet 1/13 ip address 100.40.0.2 255.255.255.0                   

       interface ethernet 1/15 ip address 100.10.0.2 255.255.255.0                   

       interface ethernet 1/25 ip address 100.50.0.2 255.255.255.0                   

       interface loopback 0 ip address 10.0.0.22 255.255.255.255                     

                                                                                     

    ##                                                                               

    ## OSPF configuration                                                            

    ##                                                                               

       protocol ospf                                                                 

       router ospf 1 vrf default                                                     

       router ospf 1 vrf default router-id 10.0.0.22                                 

       interface ethernet 1/13 ip ospf area 0.0.0.0                                  

       interface ethernet 1/15 ip ospf area 0.0.0.0                                  

       interface ethernet 1/25 ip ospf area 0.0.0.0

       interface loopback 0 ip ospf area 0.0.0.0

     

    ##

    ## BGP configuration

    ##

       protocol bgp

       router bgp 65022 vrf default

       router bgp 65022 vrf default router-id 10.0.0.22 force

       router bgp 65022 vrf default neighbor 10.0.0.11 remote-as 65011

       router bgp 65022 vrf default neighbor 10.0.0.12 remote-as 65012

       router bgp 65022 vrf default neighbor 10.0.0.13 remote-as 65013

       router bgp 65022 vrf default neighbor 10.0.0.11 update-source loopback 0

       router bgp 65022 vrf default neighbor 10.0.0.12 update-source loopback 0

       router bgp 65022 vrf default neighbor 10.0.0.13 update-source loopback 0

     

    Leaf-1 configuration

    ## MLAG protocol

    ##    

       protocol mlag

          

    ##    

    ## Interface Ethernet configuration

    ##                       

       interface mlag-port-channel 1

       interface port-channel 4000

       interface ethernet 1/17 mlag-channel-group 1 mode active

       interface ethernet 1/25-1/26 channel-group 4000 mode active                      

       interface mlag-port-channel 1 no shutdown        

                                                        

    ##                                                  

    ## LAG configuration                                

    ##                                                  

       lacp                                             

       interface ethernet 1/17 lacp rate fast           

                                                        

    ##                                                  

    ## VLAN configuration                               

    ##                                                  

       vlan 172                                         

       vlan 4000                                        

       interface mlag-port-channel 1 switchport access vlan 172

                                                        

    ##                                                  

    ## STP configuration                                

    ##                                                  

    no spanning-tree                                    

                                                        

    ##                                                  

    ## L3 configuration                                 

    ##                                                  

       ip routing vrf default                           

       interface ethernet 1/9 no switchport force       

       interface ethernet 1/13 no switchport force      

       interface loopback 0                             

       interface vlan 172                               

       interface vlan 4000                              

       interface ethernet 1/9 ip address 100.20.0.1 255.255.255.0

       interface ethernet 1/13 ip address 100.40.0.1 255.255.255.0

       interface loopback 0 ip address 10.0.0.12 255.255.255.255

       interface vlan 172 ip address 172.16.1.253 255.255.255.0

       interface vlan 4000 ip address 1.1.1.2 255.255.255.252

                                                        

    ##                                                  

    ## OSPF configuration                               

    ##                                                  

       protocol ospf                                    

       router ospf 1 vrf default                        

       router ospf 1 vrf default router-id 10.0.0.12    

       interface ethernet 1/9 ip ospf area 0.0.0.0      

       interface ethernet 1/13 ip ospf area 0.0.0.0     

       interface loopback 0 ip ospf area 0.0.0.0        

                                                        

    ##                                                  

    ## MAGP configuration                               

    ##                                                  

       protocol magp                                    

       interface vlan 172 magp 172                      

       interface vlan 172 magp 172 ip virtual-router address 172.16.1.254

       interface vlan 172 magp 172 ip virtual-router mac-address AA:BB:CC:DD:EE:FF

                                                                        

    ##                                                                  

    ## MLAG configurations                                              

    ##                                                                  

       mlag-vip MLAGCLUSTER ip 123.123.123.123 /30 force                

    no mlag shutdown                                                    

       interface port-channel 4000 ipl 1                                

       interface vlan 4000 ipl 1 peer-address 1.1.1.1                   

                                                                        

    ##                                                                  

    ## BGP configuration                                                

    ##                                                                  

       protocol bgp

       router bgp 65012 vrf default

       router bgp 65012 vrf default router-id 10.0.0.12 force

       router bgp 65012 vrf default maximum-paths 2

       router bgp 65012 vrf default bestpath as-path multipath-relax force

       router bgp 65012 vrf default neighbor 10.0.0.21 remote-as 65021

       router bgp 65012 vrf default neighbor 10.0.0.22 remote-as 65022

       router bgp 65012 vrf default neighbor 10.0.0.21 update-source loopback 0

       router bgp 65012 vrf default neighbor 10.0.0.22 update-source loopback 0

       router bgp 65012 vrf default network 172.16.1.0 /24

     

    Leaf-2 configuration

    ## MLAG protocol                

    ##                              

       protocol mlag                

                                    

    ##                              

    ## Interface Ethernet configuration

    ##                              

       interface mlag-port-channel 1

       interface port-channel 4000  

       interface ethernet 1/1 mlag-channel-group 1 mode active

       interface ethernet 1/13-1/14 channel-group 4000 mode active     

       interface mlag-port-channel 1 no shutdown        

                                                        

    ##                                                  

    ## LAG configuration                                

    ##                                                  

       lacp                                             

       interface ethernet 1/1 lacp rate fast            

                                                        

    ##                                                  

    ## VLAN configuration                               

    ##                                                  

       vlan 172                                         

       vlan 4000                                        

       interface mlag-port-channel 1 switchport access vlan 172

                                                        

    ##                                                  

    ## STP configuration                                

    ##                                                  

    no spanning-tree                                    

                                                        

    ##                                                  

    ## L3 configuration                                 

    ##                                                  

       ip routing vrf default                           

       interface ethernet 1/5 no switchport force       

       interface ethernet 1/8 no switchport force       

       interface loopback 0                             

       interface vlan 172                               

       interface vlan 4000                              

       interface ethernet 1/5 ip address 100.50.0.1 255.255.255.0

       interface ethernet 1/8 ip address 100.30.0.1 255.255.255.0

       interface loopback 0 ip address 10.0.0.11 255.255.255.255

       interface vlan 172 ip address 172.16.1.252 255.255.255.0

       interface vlan 4000 ip address 1.1.1.1 255.255.255.252

                                                                                                                                                      

    ##                                                  

    ## OSPF configuration                               

    ##                                                  

       protocol ospf                                    

       router ospf 1 vrf default                        

       router ospf 1 vrf default router-id 10.0.0.11    

       interface ethernet 1/5 ip ospf area 0.0.0.0      

       interface ethernet 1/8 ip ospf area 0.0.0.0      

       interface loopback 0 ip ospf area 0.0.0.0        

                                                        

    ##                                                  

    ## MAGP configuration                               

    ##                                                  

       protocol magp                                    

       interface vlan 172 magp 172                      

       interface vlan 172 magp 172 ip virtual-router address 172.16.1.254

       interface vlan 172 magp 172 ip virtual-router mac-address AA:BB:CC:DD:EE:FF

                                                                        

    ##                                                                  

    ## MLAG configurations                                              

    ##                                                                  

       mlag-vip MLAGCLUSTER ip 123.123.123.123 /30 force                

    no mlag shutdown                                                    

       interface port-channel 4000 ipl 1                                

       interface vlan 4000 ipl 1 peer-address 1.1.1.2                   

                                                                        

    ##                                                                  

    ## BGP configuration                                                

    ##                                                                  

       protocol bgp

       router bgp 65011 vrf default

       router bgp 65011 vrf default router-id 10.0.0.11 force

       router bgp 65011 vrf default maximum-paths 2

       router bgp 65011 vrf default bestpath as-path multipath-relax force

       router bgp 65011 vrf default neighbor 10.0.0.21 remote-as 65021

       router bgp 65011 vrf default neighbor 10.0.0.22 remote-as 65022

       router bgp 65011 vrf default neighbor 10.0.0.21 update-source loopback 0

       router bgp 65011 vrf default neighbor 10.0.0.22 update-source loopback 0

       router bgp 65011 vrf default network 172.16.1.0 /24

     

    Leaf-3 configuration

    ## L3 configuration                     

    ##                                      

       ip routing vrf default               

       interface ethernet 1/5 no switchport force

       interface ethernet 1/25 no switchport force

       interface ethernet 1/49 no switchport force

       interface loopback 0                   

       interface ethernet 1/5 ip address 100.0.0.1 255.255.255.0

       interface ethernet 1/25 ip address 100.10.0.1 255.255.255.0

       interface ethernet 1/49 ip address 172.16.2.254 255.255.255.0

       interface loopback 0 ip address 10.0.0.13 255.255.255.255

                                                                

    ##                                                          

    ## OSPF configuration                                       

    ##                                                          

       protocol ospf                                            

       router ospf 1 vrf default                                

       router ospf 1 vrf default router-id 10.0.0.13            

       interface ethernet 1/5 ip ospf area 0.0.0.0              

       interface ethernet 1/25 ip ospf area 0.0.0.0             

       interface loopback 0 ip ospf area 0.0.0.0

     

    ##

    ## BGP configuration

    ##

       protocol bgp

       router bgp 65013 vrf default

       router bgp 65013 vrf default router-id 10.0.0.13 force

       router bgp 65013 vrf default maximum-paths 2

       router bgp 65013 vrf default bestpath as-path multipath-relax force

       router bgp 65013 vrf default neighbor 10.0.0.21 remote-as 65021

       router bgp 65013 vrf default neighbor 10.0.0.22 remote-as 65022

       router bgp 65013 vrf default neighbor 10.0.0.21 update-source loopback 0

       router bgp 65013 vrf default neighbor 10.0.0.22 update-source loopback 0

       router bgp 65013 vrf default network 172.16.2.0 /24

    -- end --