EVPN Symmetric Routing with Mellanox Switches

Version 1

    This document will explain how to deploy EVPN Symmetric routing as the overlay technology on top of a leaf/spine network,

    we will be using Cumulus OS running on top of a Mellanox Spectrum switch.

     

    Topology:

     

     

    Let’s start with the basics, the requirement is to build a data center environment with the building blocks of leaves and spines.

    The data center should be connected via dark fiber to a DR site and have connectivity to the WAN,

    some of the racks will be connected with MLAG for high availability. Servers should be able to communicate via L2 across racks.

    We are going to use VXLAN in order of having L2 connectivity across racks, VXLAN will act as an overlay technology on top of our L3 clos fabric that will use BGP.

    Symmetric routing will enable routing between VXLANs.

     

     

     

     

    We are going to use EVPN as the control plane for our VXLAN deployment, our VTEPs will discover each other via EVPN and will advertise all the information to each other via the EVPN protocol.

    Once a server needs to communicate between VLANs it will be able to do so by using Symmetric routing.

     

    The reason we use Symmetric VXLAN routing is to enable scalability and efficiency.

    With symmetric routing, the ingress VTEP doesnt need to know the destination VNI for inter-VNI routing. Therefore, VTEPs dont need to learn and maintain MAC address information for the remote hosts attached to egress VNIs for which they dont have local hosts.

    This approach results in better utilization of the MAC address table and ARP adjacencies on a VTEP.

     

     

     

    As our data center is connected to the WAN and there is a requirement to advertise some routes from the WAN into our datacenter we will be using EVPN Type-5 Routes, we will inject routes from the VTEP connected towards the WAN Router into our EVPN Domain.

     

     

     

     

    Configuration Flow

     

    *We will use leaf1 as an example, full setup configuration can be found at the end.

     

    Step1

     

    First will go over the L2 configuration which in our case includes MLAG and Vlans.

     

    1.Configure MLAG between Leaf1 and Leaf2

     

    net add bond peerlink bond slaves swp7,swp8

    net add bond server1 bond slaves swp1

    net add bond server2 bond slaves swp2

     

    net add bond server1 clag id 1

    net add bond server2 clag id 2

     

    net add interface peerlink.4094 clag backup-ip 10.7.159.151

    net add interface peerlink.4094 clag peer-ip 169.254.1.2

    net add interface peerlink.4094 clag sys-mac 44:38:39:FF:40:94

    net add interface peerlink.4094 ip address 169.254.1.1/30

     

    2.Lets create an L2 bridge and add all of the VLANs we are going to use

     

    net add bridge bridge ports peerlink,server1,server2

    net add bridge bridge vids 10,20,30,55,4001

    net add bridge bridge vlan-aware

    net add bridge stp off                

     

    Step2

     

    Now we will add the SVIs that will act as gateways for our server, as we are using MLAG we will add a virtual address (VRR) for each interface that will be shared between our MLAG pair.

     

    net add vrf vrf1 vrf-table auto

    net add vlan 10 ip address 192.168.10.101/24

    net add vlan 10 ip address-virtual 00:00:00:00:00:1a 192.168.10.254/24

    net add vlan 10 vlan-id 10

    net add vlan 10 vlan-raw-device bridge

    net add vlan 10 vrf vrf1

    net add vlan 20 ip address 192.168.20.101/24

    net add vlan 20 ip address-virtual 00:00:00:00:00:2a 192.168.20.254/24

    net add vlan 20 vlan-id 20

    net add vlan 20 vlan-raw-device bridge

    net add vlan 20 vrf vrf1

    net add vlan 55 ip address 55.55.55.250/24

    net add vlan 55 ip address-virtual 00:01:02:03:04:55 55.55.55.254/24

    net add vlan 55 vlan-id 55

    net add vlan 55 vlan-raw-device bridge

    net add vlan 55 vrf vrf1

     

     

     

    Distributed Anycast Gateway

     

    In EVPN, any VTEP participating in a VNI can use the distributed anycast gateway feature for end hosts in its IP subnet by supporting the same virtual gateway IP address and the virtual gateway MAC address.

    With the anycast gateway function in EVPN, an end host in a VNI can always use its local VTEP for this VNI as its default gateway to send traffic outside its IP subnet. This capability enables optimal forwarding for northbound traffic from end hosts in the VXLAN overlay network.

    A distributed anycast gateway also offers the benefit of transparent host mobility in the VXLAN overlay network. Since the gateway IP address and virtual MAC address are identically provisioned on all VTEPs within a VNI, when an end host moves from one VTEP to another it does not need to change it's gateway address.

     

    ARP suppression

    When deploying a traditional L2 network broadcast traffic such as ARP requests creates overload on the network. By using ARP suppression with VXLAN we are able to suppress these messages at the leaf layer.

     

    As an example, At the first time Server2 communicated it sent an ARP and leaf2 learned it’s MAC and IP, Leaf2 sent an EVPN update that had the IP and MAC on the corresponding VNI4010, Leaf1 learned Server2 ip+mac on VNI4010. When Server1 sends an ARP messages to Server2 Leaf1 replies to the ARP request as it has all of the details, in this way it suppress the Broadcast to all leaves that are part of VNI4010.

     

     
    Step3

     

    BGP

     

    Now we will configure our L3 Underlay, we will be using eBGP as our underlay protocol,

     

    By using BGP unnumbered there is no need to configure the exact Ip address and the exact remote as of each neighbor, it simplifies the configuration dramatically and keeps the network agile for cable swapping’s.

     

    Unnumbered L3 interfaces are interfaces without unique IP addresses. In BGP, you configure unnumbered interfaces using Extended Next Hop Encoding (ENHE), which is defined by RFC 5549. BGP unnumbered interfaces provide a means of advertising an IPv4 route with an IPv6 next-hop.

    Prior to RFC 5549, an IPv4 route could be advertised only with an IPv4 next-hop.

    BGP unnumbered interfaces are particularly useful in deployments where IPv4 prefixes are advertised through BGP over a section without any IPv4 address configuration on links. As a result, the routing entries are also IPv4 for destination lookup and have IPv6 next-hops for forwarding purposes.

     

    Our AS design will be as follows, each of the leaf switches will be in a separate AS and the spine layer will be in the same AS layer, this is the common design for eBGP running over leaf/spine data centers.

     

     

     

    *In the following configuration, we used peer group so we can configure all of the BGP configuration in it and apply it to the interfaces.

     

    net add bgp autonomous-system 65001

    net add bgp router-id 10.0.0.1

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor FABRIC peer-group

    net add bgp neighbor FABRIC remote-as external

    net add bgp neighbor FABRIC capability extended-nexthop

    net add bgp neighbor swp9 interface peer-group FABRIC

    net add bgp neighbor swp10 interface peer-group FABRIC

    net add bgp ipv4 unicast network 10.0.0.1/32

    net add bgp ipv4 unicast network 10.10.10.20/32

    net add bgp ipv6 unicast neighbor FABRIC activate

     

     

    Step4

     

    Let’s add vlan4001 that will represent our L3 Routed VNI.

     

    L3 VNI

     

    net add vlan 4001 vlan-id 4001

    net add vlan 4001 vlan-raw-device bridge

    net add vlan 4001 vrf vrf1

    net add vxlan vxlan4001 bridge access 4001

    net add vxlan vxlan4001 vxlan id 104001

    net add vrf vrf1 vni 104001

    net add vlan 4001 hwaddress 44:39:39:FF:40:94

     

    *When two VTEPs are operating in VXLAN active-active (MLAG) mode and performing symmetric routing, you need to configure the router MAC corresponding to each layer-3 VNI to ensure both VTEPs use the same MAC address. Use the same address on both switches in the MLAG pair.

    Cumulus Networks recommends you use the MLAG system MAC address.

     

     

    VXLAN

     

    net add loopback lo clag vxlan-anycast-ip 10.10.10.20

    net add loopback lo ip address 10.0.0.1/32

    net add vxlan vtep10 bridge access 10

    net add vxlan vtep10 vxlan id 10010

    net add vxlan vtep20 bridge access 20

    net add vxlan vtep20 vxlan id 10020

    net add vxlan vtep30 bridge access 30

    net add vxlan vtep30 vxlan id 10030

    net add vxlan vtep55 bridge access 55

    net add vxlan vtep55 vxlan id 10055

    net add vxlan vtep10,20,30,55,vxlan4001 bridge arp-nd-suppress on

    net add vxlan vtep10,20,30,55,vxlan4001 bridge learning off

    net add vxlan vtep10,20,30,55,vxlan4001 mtu 9216

    net add vxlan vtep10,20,30,55,vxlan4001 stp bpduguard

    net add vxlan vtep10,20,30,55,vxlan4001 stp portbpdufilter

    net add vxlan vtep10,20,30,55,vxlan4001 vxlan local-tunnelip 10.0.0.1

    net add bridge bridge ports vtep10,vtep20,vtep30,vtep55,vxlan4001

     

    *vxlan4001 is the L3 Routed VNI, this is used to enable Symmetric routing, this VNI should be configured on all of our VTEPs.

    *clag vxlan-anycast-ip is used when we are using MLAG with VXLAN, this way other VTEPs will see our MLAG pair as a single entity.

     

    EVPN

     

    net add bgp l2vpn evpn  neighbor FABRIC activate

    net add bgp l2vpn evpn  advertise-all-vni

    * net add bgp l2vpn evpn  advertise-default-gw

     

    *advertise-default-gw is used here as we are going to use DHCP-Relay, this command is typically not used in Distributed Anycast Gateway deployments. The command will force the switch to advertise the IP and MAC of our Vlan interfaces into the EVPN Domain.

    *When using DHCP Relay it is currently required to have unique address for SVI per switch that requires use of DHCP-Relay.

     

    Step5

     

    DHCP Relay

     

    We want our Servers to get IP address via DHCP, in our case the server is located in a different VNI.

    DHCP-relay will be used in that case.

     

    Here is a suggested way for configuring the DHCP-Relay:

     

    As we are working with VRFs we will need to configure the DHCP-Relay agent inside the VRF.

     

    1. Create a file called isc-dhcp-relay-vrf1 in /etc/default/ folder, which contains the DHCP relay configuration as below:

     

         *our DHCP server is 192.168.20.4 and the local vlan we need to provide DHCP to is vlan55.

     

     

    # What servers should the DHCP relay forward requests to?

    SERVERS="192.168.20.4"

     

    # On what interfaces should the DHCP relay (dhrelay) serve DHCP requests?

    # Always include the interface towards the DHCP server.

    # This variable requires a -i for each interface configured above.

    # This will be used in the actual dhcrelay command

    # For example, "-i eth0 -i eth1"

    INTF_CMD="-i vlan55"

     

     

    2. Enable and start the service

    systemctl enable dhcrelay@vrf1.service

     

    systemctl start dhcrelay@vrf1.service

     

     

    3.Verify the service is running correctly

     

    systemctl status dhcrelay@vrf1.service

     

     

    Configuring the VTEP connected to the WAN

     

     

     

    As we need to advertise our Underlay routing into the EVPN domain we will add a BGP instance on the border leaf connected to the WAN.

     

    net add bgp autonomous-system 65004

    net add bgp router-id 10.0.0.4

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor FABRIC peer-group

    net add bgp neighbor FABRIC remote-as external

    net add bgp neighbor FABRIC capability extended-nexthop

    net add bgp neighbor swp53 interface peer-group FABRIC

    net add bgp neighbor swp54 interface peer-group FABRIC

    net add bgp ipv4 unicast network 10.0.0.4/32

    net add bgp ipv4 unicast redistribute static

    net add bgp ipv6 unicast neighbor FABRIC activate

    net add bgp l2vpn evpn  neighbor FABRIC activate

    net add bgp l2vpn evpn  advertise-all-vni

    net add bgp l2vpn evpn  advertise ipv4 unicast

     

    net add bgp vrf vrf1 autonomous-system 65004

    net add bgp vrf vrf1 router-id 10.0.0.4

    net add bgp vrf vrf1 ipv4 unicast redistribute ospf

    net add bgp vrf vrf1 l2vpn evpn  advertise ipv4 unicast

     

    Controlling Which RIB Routes Are Injected into EVPN

    By default, when announcing IP prefixes in the BGP RIB as EVPN type-5 routes, all routes in the BGP RIB are picked for advertisement as EVPN type-5 routes. You can use a route map to allow selective advertisement of routes from the BGP RIB as EVPN type-5 routes.

    The following command binds a route map filter to IPv4 EVPN type-5 route advertisement:

    net add bgp vrf vrf1 l2vpn evpn  advertise ipv4 unicast route-map map1

     

     

    Originating Default EVPN Type-5 Routes

    Cumulus Linux supports originating EVPN default type-5 routes. The default type-5 route is originated from a border (exit) leaf and advertised to all the other leafs within the pod. Any leaf within the pod follows the default route towards the border leaf for all external traffic (towards the Internet or a different pod).

    To originate a default type-5 route in EVPN, you need to execute FRRouting commands. The following shows an example:

    sudo vtysh

    switch(config)# router bgp 65004 vrf vrf1

    switch(config-router)# address-family l2vpn evpn

    switch(config-router-af)# default-originate ipv4

    switch(config-router-af)# default-originate ipv6

     

    EVPN Show Commands

     

    net show bgp l2vpn evpn summary

    Display the BGP peers participating in the layer 2 EVPN address-family and their states.

     

    net show evpn vni

    Display the configured VNIs on a network device participating in BGP EVPN. This command is only relevant on a VTEP. If symmetric routing is configured, this command displays the special layer 3 VNIs that are configured per tenant VRF.

     

    net show evpn vni 10100

    Display the EVPN information for a specific VNI in detail.

     

    net show evpn mac vni 10100 / all

    Display all local and remote MAC addresses for a VNI.

     

    net show evpn arp-cache vni 10100 / all

    Display all local and remote neighbors (ARP entries) for a VNI. This command is only relevant for a layer 2 VNI and the output shows both IPv4 and IPv6 neighbor entries.

     

    net show bgp l2vpn evpn route

    Display all EVPN routes, both local and remote. The routes displayed here are based on RD as they are across VNIs and VRFs

     

    net show route vrf vrf1

    Display all routes received via the L3 VNI.

     

     
     

     

    Full Setup configuration:

     

     

    Leaf 1 configuration:

     

    net add bond peerlink bond slaves swp7,swp8

    net add bond server1b1 bond slaves swp1

    net add bond server1b2 bond slaves swp2

    net add bond server2b1 bond slaves swp3

    net add bond server2b2 bond slaves swp4

    net add bond peerlink,server1b1-2,server2b1-2 mtu 9216

    net add bond server1b1 clag id 1

    net add bond server1b2 clag id 2

    net add bond server2b1 clag id 3

    net add bond server2b2 clag id 4

    net add bridge bridge ports peerlink,server1b1,server1b2,server2b1,server2b2,vtep10,vtep20,vtep30,vtep55,vxlan4001

    net add bridge bridge vids 10,20,30,55,4001

    net add bridge bridge vlan-aware

    net add bridge stp off

    net add interface eth0 ip address dhcp

    net add interface peerlink.4094 clag backup-ip 10.7.159.151

    net add interface peerlink.4094 clag peer-ip 169.254.1.2

    net add interface peerlink.4094 clag sys-mac 44:38:39:FF:40:94

    net add interface peerlink.4094 ip address 169.254.1.1/30

    net add interface swp1-4,7-10 mtu 9216

    net add interface swp5-6,11-16,vlan55-v0

    net add loopback lo clag vxlan-anycast-ip 10.10.10.20

    net add loopback lo ip address 10.0.0.1/32

    net add vlan 10 ip address 192.168.10.101/24

    net add vlan 10 ip address-virtual 00:00:00:00:00:1a 192.168.10.254/24

    net add vlan 10 vlan-id 10

    net add vlan 10 vlan-raw-device bridge

    net add vlan 10 vrf vrf1

    net add vlan 20 ip address 192.168.20.101/24

    net add vlan 20 ip address-virtual 00:00:00:00:00:2a 192.168.20.254/24

    net add vlan 20 vlan-id 20

    net add vlan 20 vlan-raw-device bridge

    net add vlan 20 vrf vrf1

    net add vlan 4001 hwaddress 44:39:39:FF:40:94

    net add vlan 4001 vlan-id 4001

    net add vlan 4001 vlan-raw-device bridge

    net add vlan 4001 vrf vrf1

    net add vlan 55 ip address 55.55.55.250/24

    net add vlan 55 ip address-virtual 00:01:02:03:04:55 55.55.55.254/24

    net add vlan 55 vlan-id 55

    net add vlan 55 vlan-raw-device bridge

    net add vlan 55 vrf vrf1

    net add vrf vrf1 vrf-table auto

    net add vxlan vtep10 bridge access 10

    net add vxlan vtep10 vxlan id 10010

    net add vxlan vtep10,20,30,55,vxlan4001 bridge arp-nd-suppress on

    net add vxlan vtep10,20,30,55,vxlan4001 bridge learning off

    net add vxlan vtep10,20,30,55,vxlan4001 mtu 9216

    net add vxlan vtep10,20,30,55,vxlan4001 stp bpduguard

    net add vxlan vtep10,20,30,55,vxlan4001 stp portbpdufilter

    net add vxlan vtep10,20,30,55,vxlan4001 vxlan local-tunnelip 10.0.0.1

    net add vxlan vtep20 bridge access 20

    net add vxlan vtep20 vxlan id 10020

    net add vxlan vtep30 bridge access 30

    net add vxlan vtep30 vxlan id 10030

    net add vxlan vtep55 bridge access 55

    net add vxlan vtep55 vxlan id 10055

    net add vxlan vxlan4001 bridge access 4001

    net add vxlan vxlan4001 vxlan id 104001

    net add vrf vrf1 vni 104001

    net add interface swp9-10 ipv6 nd ra-interval 10

    net del interface swp9-10 ipv6 nd suppress-ra

    net add routing defaults datacenter

    net add routing service integrated-vtysh-config

    net add routing log syslog informational

    net add bgp autonomous-system 65001

    net add bgp router-id 10.0.0.1

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor FABRIC peer-group

    net add bgp neighbor FABRIC remote-as external

    net add bgp neighbor FABRIC capability extended-nexthop

    net add bgp neighbor swp9 interface peer-group FABRIC

    net add bgp neighbor swp10 interface peer-group FABRIC

    net add bgp ipv4 unicast network 10.0.0.1/32

    net add bgp ipv4 unicast network 10.10.10.20/32

    net add bgp ipv6 unicast neighbor FABRIC activate

    net add bgp l2vpn evpn  neighbor FABRIC activate

    net add bgp l2vpn evpn  advertise-all-vni

    net add bgp l2vpn evpn  advertise-default-gw

     

     

    Leaf2 Configuration:

     

    net add bond peerlink bond slaves swp7,swp8

    net add bond server1b1 bond slaves swp1

    net add bond server1b2 bond slaves swp2

    net add bond server2b1 bond slaves swp3

    net add bond server2b2 bond slaves swp4

    net add bond peerlink,server1b1-2,server2b1-2 mtu 9216

    net add bond server1b1 clag id 1

    net add bond server1b2 clag id 2

    net add bond server2b1 clag id 3

    net add bond server2b2 clag id 4

    net add bridge bridge ports peerlink,server1b1,server1b2,server2b1,server2b2,vtep10,vtep20,vtep30,vtep55,vxlan4001

    net add bridge bridge vids 10,20,30,55,4001

    net add bridge bridge vlan-aware

    net add bridge stp off

    net add interface eth0 ip address dhcp

    net add interface peerlink.4094 clag backup-ip 10.7.159.150

    net add interface peerlink.4094 clag peer-ip 169.254.1.1

    net add interface peerlink.4094 clag sys-mac 44:38:39:FF:40:94

    net add interface peerlink.4094 ip address 169.254.1.2/30

    net add interface swp1-4,7-10 mtu 9216

    net add interface swp5-6,11-16,vlan20-v0,vlan55-v0

    net add loopback lo clag vxlan-anycast-ip 10.10.10.20

    net add loopback lo ip address 10.0.0.2/32

    net add vlan 10 ip address 192.168.10.102/24

    net add vlan 10 ip address-virtual 00:00:00:00:00:1a 192.168.10.254/24

    net add vlan 10 vlan-id 10

    net add vlan 10 vlan-raw-device bridge

    net add vlan 10 vrf vrf1

    net add vlan 20 ip address 192.168.20.102/24

    net add vlan 20 ip address-virtual 00:00:00:00:00:2a 192.168.20.254/24

    net add vlan 20 vlan-id 20

    net add vlan 20 vlan-raw-device bridge

    net add vlan 20 vrf vrf1

    net add vlan 4001 hwaddress 44:39:39:FF:40:94

    net add vlan 4001 vlan-id 4001

    net add vlan 4001 vlan-raw-device bridge

    net add vlan 4001 vrf vrf1

    net add vlan 55 ip address 55.55.55.249/24

    net add vlan 55 ip address-virtual 00:01:02:03:04:55 55.55.55.254/24

    net add vlan 55 vlan-id 55

    net add vlan 55 vlan-raw-device bridge

    net add vlan 55 vrf vrf1

    net add vrf vrf1 vrf-table auto

    net add vxlan vtep10 bridge access 10

    net add vxlan vtep10 vxlan id 10010

    net add vxlan vtep10,20,30,55,vxlan4001 bridge arp-nd-suppress on

    net add vxlan vtep10,20,30,55,vxlan4001 bridge learning off

    net add vxlan vtep10,20,30,55,vxlan4001 mtu 9216

    net add vxlan vtep10,20,30,55,vxlan4001 stp bpduguard

    net add vxlan vtep10,20,30,55,vxlan4001 stp portbpdufilter

    net add vxlan vtep10,20,30,55,vxlan4001 vxlan local-tunnelip 10.0.0.2

    net add vxlan vtep20 bridge access 20

    net add vxlan vtep20 vxlan id 10020

    net add vxlan vtep30 bridge access 30

    net add vxlan vtep30 vxlan id 10030

    net add vxlan vtep55 bridge access 55

    net add vxlan vtep55 vxlan id 10055

    net add vxlan vxlan4001 bridge access 4001

    net add vxlan vxlan4001 vxlan id 104001

    net add vrf vrf1 vni 104001

    net add hostname cumulus

    net add dhcp relay server 192.168.20.4

    net add dhcp relay interface vlan55,vlan55-v0

    net add interface swp1-32 breakout 1x

    net add interface swp9-10 ipv6 nd ra-interval 10

    net del interface swp9-10 ipv6 nd suppress-ra

    net add routing defaults datacenter

    net add routing service integrated-vtysh-config

    net add routing log syslog informational

    net add bgp autonomous-system 65002

    net add bgp router-id 10.0.0.2

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor FABRIC peer-group

    net add bgp neighbor FABRIC remote-as external

    net add bgp neighbor FABRIC capability extended-nexthop

    net add bgp neighbor swp9 interface peer-group FABRIC

    net add bgp neighbor swp10 interface peer-group FABRIC

    net add bgp ipv4 unicast network 10.0.0.2/32

    net add bgp ipv4 unicast network 10.10.10.20/32

    net add bgp ipv6 unicast neighbor FABRIC activate

    net add bgp l2vpn evpn  neighbor FABRIC activate

    net add bgp l2vpn evpn  advertise-all-vni

    net add bgp l2vpn evpn  advertise-default-gw

     

     

     

    Leaf3 Configuration:

     

    net add bridge bridge ports swp49,swp50,swp51,swp52,vtep10,vtep20,vtep30,vxlan4001

    net add bridge bridge vids 10,20,30,4001

    net add bridge bridge vlan-aware

    net add bridge stp off

    net add interface eth0 ip address dhcp

    net add interface swp1-16,53-54

    net add interface swp49-52,55-56 mtu 9216

    net add loopback lo ip address 10.0.0.3/32

    net add vlan 10 ip address 192.168.10.103/24

    net add vlan 10 ip address-virtual 00:00:00:00:00:1a 192.168.10.254/24

    net add vlan 10 vlan-id 10

    net add vlan 10 vlan-raw-device bridge

    net add vlan 10 vrf vrf1

    net add vlan 20 ip address 192.168.20.103/24

    net add vlan 20 ip address-virtual 00:00:00:00:00:2a 192.168.20.254/24

    net add vlan 20 vlan-id 20

    net add vlan 20 vlan-raw-device bridge

    net add vlan 20 vrf vrf1

    net add vlan 4001 vlan-id 4001

    net add vlan 4001 vlan-raw-device bridge

    net add vlan 4001 vrf vrf1

    net add vrf vrf1 vrf-table auto

    net add vxlan vtep10 bridge access 10

    net add vxlan vtep10 vxlan id 10010

    net add vxlan vtep10,20,30,vxlan4001 bridge arp-nd-suppress on

    net add vxlan vtep10,20,30,vxlan4001 bridge learning off

    net add vxlan vtep10,20,30,vxlan4001 mtu 9216

    net add vxlan vtep10,20,30,vxlan4001 stp bpduguard

    net add vxlan vtep10,20,30,vxlan4001 stp portbpdufilter

    net add vxlan vtep10,20,30,vxlan4001 vxlan local-tunnelip 10.0.0.3

    net add vxlan vtep20 bridge access 20

    net add vxlan vtep20 vxlan id 10020

    net add vxlan vtep30 bridge access 30

    net add vxlan vtep30 vxlan id 10030

    net add vxlan vxlan4001 bridge access 4001

    net add vxlan vxlan4001 vxlan id 104001

    net add vrf vrf1 vni 104001

    net add hostname cumulus

    net add interface swp1-56 breakout 1x

    net add interface swp55-56 ipv6 nd ra-interval 10

    net del interface swp55-56 ipv6 nd suppress-ra

    net add routing defaults datacenter

    net add routing service integrated-vtysh-config

    net add routing log syslog informational

    net add bgp autonomous-system 65003

    net add bgp router-id 10.0.0.3

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor FABRIC peer-group

    net add bgp neighbor FABRIC remote-as external

    net add bgp neighbor FABRIC capability extended-nexthop

    net add bgp neighbor swp55 interface peer-group FABRIC

    net add bgp neighbor swp56 interface peer-group FABRIC

    net add bgp ipv4 unicast network 10.0.0.3/32

    net add bgp ipv6 unicast neighbor FABRIC activate

    net add bgp l2vpn evpn  neighbor FABRIC activate

    net add bgp l2vpn evpn  advertise-all-vni

     

     

     

      

     

    Spine1 Configuration:

     

    net add interface swp1-2,6-8,11-16

    net add interface swp3-5,9-10 mtu 9216

    net add loopback lo ip address 10.0.0.11/32

    net add hostname cumulus

    net add interface swp1-16 breakout 1x

    net add interface swp3-5,9-10 ipv6 nd ra-interval 10

    net del interface swp3-5,9-10 ipv6 nd suppress-ra

    net add routing defaults datacenter

    net add routing service integrated-vtysh-config

    net add routing log syslog informational

    net add bgp autonomous-system 65010

    net add bgp router-id 10.0.0.11

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor FABRIC peer-group

    net add bgp neighbor FABRIC remote-as external

    net add bgp neighbor FABRIC capability extended-nexthop

    net add bgp neighbor swp3 interface peer-group FABRIC

    net add bgp neighbor swp4 interface peer-group FABRIC

    net add bgp neighbor swp5 interface peer-group FABRIC

    net add bgp neighbor swp9 interface peer-group FABRIC

    net add bgp neighbor swp10 interface peer-group FABRIC

    net add bgp ipv4 unicast network 10.0.0.11/32

    net add bgp ipv6 unicast neighbor FABRIC activate

    net add bgp l2vpn evpn  neighbor FABRIC activate

    net add bgp l2vpn evpn  advertise-all-vni

     

    Spine2 Configuration:

     

    net add interface swp1-2,5,7-8,11-16

    net add interface swp3-4,6,9-10 mtu 9216

    net add loopback lo ip address 10.0.0.12/32

    net add hostname cumulus

    net add interface swp1-16 breakout 1x

    net add interface swp3-4,6,9-10 ipv6 nd ra-interval 10

    net del interface swp3-4,6,9-10 ipv6 nd suppress-ra

    net add routing defaults datacenter

    net add routing service integrated-vtysh-config

    net add routing log syslog informational

    net add bgp autonomous-system 65010

    net add bgp router-id 10.0.0.12

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor FABRIC peer-group

    net add bgp neighbor FABRIC remote-as external

    net add bgp neighbor FABRIC capability extended-nexthop

    net add bgp neighbor swp3 interface peer-group FABRIC

    net add bgp neighbor swp4 interface peer-group FABRIC

    net add bgp neighbor swp6 interface peer-group FABRIC

    net add bgp neighbor swp9 interface peer-group FABRIC

    net add bgp neighbor swp10 interface peer-group FABRIC

    net add bgp ipv4 unicast network 10.0.0.12/32

    net add bgp ipv6 unicast neighbor FABRIC activate

    net add bgp l2vpn evpn  neighbor FABRIC activate

    net add bgp l2vpn evpn  advertise-all-vni

     

    Border leaf1 (WAN OSPF) Configuration:

     

    net add bridge bridge ports swp52,vxlan4001

    net add bridge bridge vids 45,4001

    net add bridge bridge vlan-aware

    net add bridge stp off

    net add interface eth0 ip address dhcp

    net add interface swp1-16,49-51,55-56

    net add interface swp52 bridge access 45

    net add interface swp52-54 mtu 9216

    net add loopback lo ip address 10.0.0.4/32

    net add vlan 4001 vlan-id 4001

    net add vlan 4001 vlan-raw-device bridge

    net add vlan 4001 vrf vrf1

    net add vlan 45 ip address 45.45.45.1/24

    net add vlan 45 mtu 9212

    net add vlan 45 vlan-id 45

    net add vlan 45 vlan-raw-device bridge

    net add vlan 45 vrf vrf1

    net add vrf vrf1 vrf-table auto

    net add vxlan vxlan4001 bridge access 4001

    net add vxlan vxlan4001 bridge arp-nd-suppress on

    net add vxlan vxlan4001 bridge learning off

    net add vxlan vxlan4001 mtu 9216

    net add vxlan vxlan4001 stp bpduguard

    net add vxlan vxlan4001 stp portbpdufilter

    net add vxlan vxlan4001 vxlan id 104001

    net add vxlan vxlan4001 vxlan local-tunnelip 10.0.0.4

    net add vrf vrf1 vni 104001

    net add hostname cumulus

    net add interface swp1-56 breakout 1x

    net add interface swp53-54 ipv6 nd ra-interval 10

    net del interface swp53-54 ipv6 nd suppress-ra

    net add routing defaults datacenter

    net add routing service integrated-vtysh-config

    net add routing log syslog informational

    net add bgp autonomous-system 65004

    net add bgp router-id 10.0.0.4

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor FABRIC peer-group

    net add bgp neighbor FABRIC remote-as external

    net add bgp neighbor FABRIC capability extended-nexthop

    net add bgp neighbor swp53 interface peer-group FABRIC

    net add bgp neighbor swp54 interface peer-group FABRIC

    net add bgp ipv4 unicast network 10.0.0.4/32

    net add bgp ipv4 unicast redistribute static

    net add bgp ipv6 unicast neighbor FABRIC activate

    net add bgp l2vpn evpn  neighbor FABRIC activate

    net add bgp l2vpn evpn  advertise-all-vni

    net add bgp l2vpn evpn  advertise ipv4 unicast

    net add bgp vrf vrf1 autonomous-system 65004

    net add bgp vrf vrf1 router-id 10.0.0.4

    net add bgp vrf vrf1 ipv4 unicast network 45.45.45.0/24

    net add bgp vrf vrf1 ipv4 unicast redistribute ospf

    net add bgp vrf vrf1 l2vpn evpn  advertise ipv4 unicast

    net add ospf vrf vrf1

    net add ospf vrf vrf1 redistribute bgp

    net add ospf vrf vrf1 network 45.45.45.0/24 area 0

     

     

    Border leaf2 (DCI Main) Configuration:

     

    net add interface swp10,53-54 mtu 9216

    net add interface swp3-9,11-16,49-52,55-56

    net add loopback lo ip address 10.0.0.6/32

    net add hostname cumulus

    net add interface swp1-56 breakout 1x

    net add interface swp10,53-54 ipv6 nd ra-interval 10

    net del interface swp10,53-54 ipv6 nd suppress-ra

    net add routing defaults datacenter

    net add routing service integrated-vtysh-config

    net add routing log syslog informational

    net add bgp autonomous-system 65006

    net add bgp router-id 10.0.0.6

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor FABRIC peer-group

    net add bgp neighbor FABRIC remote-as external

    net add bgp neighbor FABRIC capability extended-nexthop

    net add bgp neighbor swp10 interface peer-group FABRIC

    net add bgp neighbor swp53 interface peer-group FABRIC

    net add bgp neighbor swp54 interface peer-group FABRIC

    net add bgp ipv4 unicast network 10.0.0.6/32

    net add bgp ipv6 unicast neighbor FABRIC activate

    net add bgp l2vpn evpn  neighbor FABRIC activate

    net add bgp l2vpn evpn  advertise-all-vni

     

     

    Border leaf3 (DCI DR) Configuration:

     

    net add interface swp10,49 mtu 9216

    net add interface swp3-9,11-16,50-56

    net add loopback lo ip address 10.0.0.7/32

    net add hostname cumulus

    net add interface swp1-56 breakout 1x

    net add interface swp10,49 ipv6 nd ra-interval 10

    net del interface swp10,49 ipv6 nd suppress-ra

    net add routing defaults datacenter

    net add routing service integrated-vtysh-config

    net add routing log syslog informational

    net add bgp autonomous-system 65007

    net add bgp router-id 10.0.0.7

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor FABRIC peer-group

    net add bgp neighbor FABRIC remote-as external

    net add bgp neighbor FABRIC capability extended-nexthop

    net add bgp neighbor swp10 interface peer-group FABRIC

    net add bgp neighbor swp49 interface peer-group FABRIC

    net add bgp ipv4 unicast network 10.0.0.7/32

    net add bgp ipv6 unicast neighbor FABRIC activate

    net add bgp l2vpn evpn  neighbor FABRIC activate

    net add bgp l2vpn evpn  advertise-all-vni

     

     

    leaf4 (DR site leaf) Configuration:

     

    net add bridge bridge ports swp1,vtep10,vtep20,vtep30,vtep55,vxlan4001

    net add bridge bridge vids 10,20,30,55,4001

    net add bridge bridge vlan-aware

    net add bridge stp off

    net add interface eth0 ip address dhcp

    net add interface swp1,9 mtu 9216

    net add interface swp2-8,10-16

    net add loopback lo ip address 10.0.0.5/32

    net add vlan 10 ip address 192.168.10.105/24

    net add vlan 10 ip address-virtual 00:00:00:00:00:1a 192.168.10.254/24

    net add vlan 10 vlan-id 10

    net add vlan 10 vlan-raw-device bridge

    net add vlan 10 vrf vrf1

    net add vlan 20 ip address 192.168.20.105/24

    net add vlan 20 ip address-virtual 00:00:00:00:00:2a 192.168.20.254/24

    net add vlan 20 vlan-id 20

    net add vlan 20 vlan-raw-device bridge

    net add vlan 20 vrf vrf1

    net add vlan 4001 vlan-id 4001

    net add vlan 4001 vlan-raw-device bridge

    net add vlan 4001 vrf vrf1

    net add vlan 55 ip address 55.55.55.252/24

    net add vlan 55 ip address-virtual 00:01:02:03:04:55 55.55.55.254/24

    net add vlan 55 vlan-id 55

    net add vlan 55 vlan-raw-device bridge

    net add vlan 55 vrf vrf1

    net add vrf vrf1 vrf-table auto

    net add vxlan vtep10 bridge access 10

    net add vxlan vtep10 vxlan id 10010

    net add vxlan vtep10,20,30,55,vxlan4001 bridge arp-nd-suppress on

    net add vxlan vtep10,20,30,55,vxlan4001 bridge learning off

    net add vxlan vtep10,20,30,55,vxlan4001 mtu 9216

    net add vxlan vtep10,20,30,55,vxlan4001 stp bpduguard

    net add vxlan vtep10,20,30,55,vxlan4001 stp portbpdufilter

    net add vxlan vtep10,20,30,55,vxlan4001 vxlan local-tunnelip 10.0.0.5

    net add vxlan vtep20 bridge access 20

    net add vxlan vtep20 vxlan id 10020

    net add vxlan vtep30 bridge access 30

    net add vxlan vtep30 vxlan id 10030

    net add vxlan vtep55 bridge access 55

    net add vxlan vtep55 vxlan id 10055

    net add vxlan vxlan4001 bridge access 4001

    net add vxlan vxlan4001 vxlan id 104001

    net add hostname cumulus

    net add interface swp1-16 breakout 1x

    net add interface swp9 ipv6 nd ra-interval 10

    net del interface swp9 ipv6 nd suppress-ra

    net add routing defaults datacenter

    net add routing service integrated-vtysh-config

    net add routing log syslog informational

    net add bgp autonomous-system 65005

    net add bgp router-id 10.0.0.5

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor FABRIC peer-group

    net add bgp neighbor FABRIC remote-as external

    net add bgp neighbor FABRIC capability extended-nexthop

    net add bgp neighbor swp9 interface peer-group FABRIC

    net add bgp ipv4 unicast network 10.0.0.5/32

    net add bgp ipv6 unicast neighbor FABRIC activate

    net add bgp l2vpn evpn  neighbor FABRIC activate

    net add bgp l2vpn evpn  advertise-all-vni