EVPN with Mellanox Switches

Version 1

    EVPN

     

    As explained RFC 7432, EVPN (stands for Ethernet Virtual Private Network) is new standard for making data center overlays which is becoming default standard on how the overlays are made in data centers, how data centers can be inter connected and how it can be done without any controller (which locks in customers with proeritary protocols).

    EVPN works as a control plane for VXLAN by distributing MAC and IP address for VTEPs (VXLAN Tunnel End Points). EVPN uses BGP to propagate the control plane information of MAC/IP. Because EVPN leverages the proven BGP protocol for control plane, it is becoming choice for making VXLANs.

    EVPN has different type of control packets (Type packets) which take care of making the control plane. Table 1.1 shows the details of such packets.

     

    Route Type

    Route Type Name

    Purpose

    Support on Mellanox Switches

    Type – 1

    Ethernet Auto-discovery Route

    • Needed for taking care of dual homes hosts
    • Auto Discovery per ESI for Fast Convergence & Spilt horizon
    • Auto Discovery per EVI route is used for Aliasing

    Need for Type – 1 route types

    Active - Active is taken care by MLAG

    EVPN interacts with MLAG

    Type – 2

    MAC/IP Route

    • Advertise host MAC and IP address as soon as hosts comes up.
    • Proactive learning suppress ARP (reducing ARP storms)

    Supported

    Type – 3

    Inclusive Multicast Route

    • Used to advertise the location of VTEPs in a VXLAN
    • Used to forward BUM traffic between VTEPs (Head end replication). This is used if the MAC is not known.

    Supported

    Type – 4

    Ethernet Segment

    • Needed for taking care of dual homed hosts
    • Route Discovery of nodes in redundancy group
    • Selection of Dedicated forwarder (DF-election)

    Need for Type – 4 route types

    Active - Active is taken care by MLAG

    EVPN interacts with MLAG

    Type – 5

    IP Prefix route

    • Advertise IP prefixes

    Supported

     

     

    Type – 1 and Type – 4 packets are used if the underlay is MPLS based. In case of IP fabric MLAG can take care of Type – 1 and Type – 4 scenarios.

     

    Topology

     

    Configurations

     

    Spine01 Configurations

     

    cumulus@spine01:mgmt-vrf:~$ net show configuration commands

    net del all

    net add interface swp31-32 alias to Spine02

    net add loopback lo ip address 10.100.100.1/32

    net add loopback lo alias loopback interface

    net add interface eth0 ip address dhcp

    net add interface eth0 alias management interface

    net add interface eth0 vrf mgmt

    net add interface swp1 alias to Leaf01

    net add interface swp2 alias to Leaf02

    net add interface swp3 alias to Leaf03

    net add interface swp4 alias to Leaf04

    net add interface swp29 alias to Exit01

    net add interface swp30 alias to Exit02

    net add vrf mgmt ip address 127.0.0.1/8

    net add vrf mgmt vrf-table auto

    net add hostname spine01

    net add interface swp1-16 breakout 1x

    net add interface swp1-4,29-30 ipv6 nd ra-interval 10

    net del interface swp1-4,29-30 ipv6 nd suppress-ra

    net add routing service integrated-vtysh-config

    net add routing log file /var/log/quagga/quagga.log

    net add routing log timestamp precision 6

    net add bgp autonomous-system 65531

    net add bgp router-id 10.100.100.1

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor swp1 interface remote-as external

    net add bgp neighbor swp2 interface remote-as external

    net add bgp neighbor swp3 interface remote-as external

    net add bgp neighbor swp4 interface remote-as external

    net add bgp neighbor swp29 interface remote-as external

    net add bgp neighbor swp30 interface remote-as external

    net add bgp ipv4 unicast network 10.100.100.1/32

    net add time zone Etc/UTC

    net add time ntp server 0.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 1.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 2.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 3.cumulusnetworks.pool.ntp.org iburst

    net add time ntp source eth0

    net add dns nameserver ipv4 10.20.4.187 vrf mgmt

    net add snmp-server listening-address localhost

     

     

    There are some configuration commands that are not yet supported by nclu.

    The following will append those commands to the appropriate files.

    =======================================================================

    printf 'frr version 3.1+cl3u2\n' >> /etc/quagga/Quagga.conf

    printf 'frr defaults datacenter\n' >> /etc/quagga/Quagga.conf

    printf 'username cumulus nopassword\n' >> /etc/quagga/Quagga.conf

    printf 'router-id 10.100.100.1\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65531\n  address-family l2vpn evpn\n  neighbor swp1 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65531\n  address-family l2vpn evpn\n  neighbor swp2 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65531\n  address-family l2vpn evpn\n  neighbor swp3 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65531\n  address-family l2vpn evpn\n  neighbor swp4 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65531\n  address-family l2vpn evpn\n  neighbor swp29 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65531\n  address-family l2vpn evpn\n  neighbor swp30 activate\n' >> /etc/quagga/Quagga.conf

    printf 'line vty\n' >> /etc/quagga/Quagga.conf

    cumulus@spine01:mgmt-vrf:~$

     

     

    Spine02 Configurations

     

     

     

    cumulus@spine02:~$ net show configuration commands

    net del all

    net add interface swp31-32 alias to Spine02

    net add loopback lo ip address 10.100.100.2/32

    net add loopback lo alias loopback interface

    net add interface eth0 ip address dhcp

    net add interface eth0 alias management interface

    net add interface eth0 vrf mgmt

    net add interface swp1 alias to Leaf01

    net add interface swp2 alias to Leaf02

    net add interface swp3 alias to Leaf03

    net add interface swp4 alias to Leaf04

    net add interface swp29 alias to Exit01

    net add interface swp30 alias to Exit02

    net add vrf mgmt ip address 127.0.0.1/8

    net add vrf mgmt vrf-table auto

    net add hostname spine02

    net add interface swp1-16 breakout 1x

    net add interface swp1-4,29-30 ipv6 nd ra-interval 10

    net del interface swp1-4,29-30 ipv6 nd suppress-ra

    net add routing service integrated-vtysh-config

    net add routing log file /var/log/quagga/quagga.log

    net add routing log timestamp precision 6

    net add bgp autonomous-system 65532

    net add bgp router-id 10.100.100.2

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor swp1 interface remote-as external

    net add bgp neighbor swp2 interface remote-as external

    net add bgp neighbor swp3 interface remote-as external

    net add bgp neighbor swp4 interface remote-as external

    net add bgp neighbor swp29 interface remote-as external

    net add bgp neighbor swp30 interface remote-as external

    net add bgp ipv4 unicast network 10.100.100.2/32

    net add time zone Etc/UTC

    net add time ntp server 0.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 1.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 2.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 3.cumulusnetworks.pool.ntp.org iburst

    net add time ntp source eth0

    net add dns nameserver ipv4 10.20.4.187 vrf mgmt

    net add dns nameserver ipv4 10.20.4.167 vrf mgmt

    net add dns nameserver ipv4 10.2.8.150 vrf mgmt

    net add dns nameserver ipv4 10.2.8.3 vrf mgmt

    net add dns nameserver ipv4 10.2.8.4 vrf mgmt

    net add snmp-server listening-address localhost

     

     

    There are some configuration commands that are not yet supported by nclu.

    The following will append those commands to the appropriate files.

    =======================================================================

    printf 'frr version 3.1+cl3u2\n' >> /etc/quagga/Quagga.conf

    printf 'frr defaults datacenter\n' >> /etc/quagga/Quagga.conf

    printf 'username cumulus nopassword\n' >> /etc/quagga/Quagga.conf

    printf 'router-id 10.100.100.2\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65532\n  address-family l2vpn evpn\n  neighbor swp1 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65532\n  address-family l2vpn evpn\n  neighbor swp2 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65532\n  address-family l2vpn evpn\n  neighbor swp3 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65532\n  address-family l2vpn evpn\n  neighbor swp4 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65532\n  address-family l2vpn evpn\n  neighbor swp29 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65532\n  address-family l2vpn evpn\n  neighbor swp30 activate\n' >> /etc/quagga/Quagga.conf

    printf 'line vty\n' >> /etc/quagga/Quagga.conf

    cumulus@spine02:~$

     

     

     

    Leaf01 Configurations

     

     

     

     

    cumulus@leaf01:mgmt-vrf:~$ net show configuration commands

    net del all

    net add interface swp15-16 alias to Leaf02

    net add vxlan vni13 vxlan id 13

    net add vxlan vni13 vxlan local-tunnelip 10.1.1.1

    net add loopback lo ip address 10.1.1.1/32

    net add loopback lo clag vxlan-anycast-ip 10.10.10.1

    net add loopback lo alias loopback interface

    net add interface eth0 ip address dhcp

    net add interface eth0 alias management interface

    net add interface eth0 vrf mgmt

    net add interface swp5 alias to Spine01

    net add interface swp6 alias to Spine02

    net add interface swp7 link autoneg on

    net add interface swp7 alias to Server

    net add bond bond01 bond slaves swp7

    net add bond bond01 bridge access 13

    net add bond bond01 clag id 1

    net add bridge bridge vlan-aware

    net add bridge bridge ports bond01,peerlink,vni13,vni24

    net add bridge bridge vids 13,24

    net add bridge bridge pvid 1

    net add vrf mgmt ip address 127.0.0.1/8

    net add vrf mgmt vrf-table auto

    net add bond peerlink bond slaves swp15,swp16

    net add interface peerlink.4094 ip address 169.254.1.1/30

    net add interface peerlink.4094 clag peer-ip 169.254.1.2

    net add interface peerlink.4094 clag backup-ip 10.1.1.2

    net add interface peerlink.4094 clag sys-mac 44:39:39:ff:40:94

    net add interface peerlink.4094 clag priority 100

    net add vxlan vni13 bridge access 13

    net add vxlan vni13 bridge learning off

    net add vxlan vni13 stp bpduguard

    net add vxlan vni13 stp portbpdufilter

    net add hostname leaf01

    net add interface swp1-16 breakout 1x

    net add interface swp5-6 ipv6 nd ra-interval 10

    net del interface swp5-6 ipv6 nd suppress-ra

    net add routing service integrated-vtysh-config

    net add routing log timestamp precision 6

    net add bgp autonomous-system 65001

    net add bgp router-id 10.1.1.1

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor swp5 interface remote-as external

    net add bgp neighbor swp6 interface remote-as external

    net add bgp ipv4 unicast network 10.1.1.1/32

    net add bgp ipv4 unicast network 10.10.10.1/32

    net add time zone Etc/UTC

    net add time ntp server 0.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 1.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 2.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 3.cumulusnetworks.pool.ntp.org iburst

    net add time ntp source eth0

    net add dns nameserver ipv4 10.20.4.187 vrf mgmt

    net add snmp-server listening-address localhost

     

     

    There are some configuration commands that are not yet supported by nclu.

    The following will append those commands to the appropriate files.

    =======================================================================

    printf 'frr version 3.1+cl3u2\n' >> /etc/quagga/Quagga.conf

    printf 'frr defaults datacenter\n' >> /etc/quagga/Quagga.conf

    printf 'username cumulus nopassword\n' >> /etc/quagga/Quagga.conf

    printf 'router-id 10.1.1.1\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65001\n  address-family l2vpn evpn\n  neighbor swp5 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65001\n  address-family l2vpn evpn\n  neighbor swp6 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65001\n  address-family l2vpn evpn\n  advertise-all-vni\n' >> /etc/quagga/Quagga.conf

    printf 'line vty\n' >> /etc/quagga/Quagga.conf

    cumulus@leaf01:mgmt-vrf:~$

     

     

     

    Leaf02 Configurations

     

     

     

     

    cumulus@leaf02:mgmt-vrf:~$ net show configuration commands

    net del all

    net add interface swp1-2 alias to Leaf01

    net add vxlan vni13 vxlan id 13

    net add vxlan vni13 vxlan local-tunnelip 10.1.1.2

    net add loopback lo ip address 10.1.1.2/32

    net add loopback lo clag vxlan-anycast-ip 10.10.10.1

    net add loopback lo alias loopback interface

    net add interface eth0 ip address dhcp

    net add interface eth0 alias management interface

    net add interface eth0 vrf mgmt

    net add interface swp5 alias to Spine01

    net add interface swp6 alias to Spine02

    net add interface swp7 link autoneg on

    net add interface swp7 alias to Server01

    net add bond bond01 bond slaves swp7

    net add bond bond01 bridge access 13

    net add bond bond01 clag id 1

    net add bridge bridge vlan-aware

    net add bridge bridge ports bond01,peerlink,vni13,vni24

    net add bridge bridge vids 13,24

    net add bridge bridge pvid 1

    net add vrf mgmt ip address 127.0.0.1/8

    net add vrf mgmt vrf-table auto

    net add bond peerlink bond slaves swp1,swp2

    net add interface peerlink.4094 ip address 169.254.1.2/30

    net add interface peerlink.4094 clag peer-ip 169.254.1.1

    net add interface peerlink.4094 clag backup-ip 10.1.1.1

    net add interface peerlink.4094 clag sys-mac 44:39:39:ff:40:94

    net add interface peerlink.4094 clag priority 200

    net add vxlan vni13 bridge access 13

    net add vxlan vni13 bridge learning off

    net add vxlan vni13 stp bpduguard

    net add vxlan vni13 stp portbpdufilter

    net add hostname leaf02

    net add interface swp1-32 breakout 1x

    net add interface swp5-6 ipv6 nd ra-interval 10

    net del interface swp5-6 ipv6 nd suppress-ra

    net add routing service integrated-vtysh-config

    net add routing log file /var/log/quagga/quagga.log

    net add routing log timestamp precision 6

    net add bgp autonomous-system 65002

    net add bgp router-id 10.1.1.2

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor swp5 interface remote-as external

    net add bgp neighbor swp6 interface remote-as external

    net add bgp ipv4 unicast network 10.1.1.2/32

    net add bgp ipv4 unicast network 10.10.10.1/32

    net add time zone Etc/UTC

    net add time ntp server 0.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 1.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 2.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 3.cumulusnetworks.pool.ntp.org iburst

    net add time ntp source eth0

    net add dns nameserver ipv4 10.20.4.187 vrf mgmt

    net add dns nameserver ipv4 10.20.4.167 vrf mgmt

    net add dns nameserver ipv4 10.2.8.150 vrf mgmt

    net add dns nameserver ipv4 10.2.8.3 vrf mgmt

    net add dns nameserver ipv4 10.2.8.4 vrf mgmt

    net add snmp-server listening-address localhost

     

     

    There are some configuration commands that are not yet supported by nclu.

    The following will append those commands to the appropriate files.

    =======================================================================

    printf 'frr version 3.1+cl3u2\n' >> /etc/quagga/Quagga.conf

    printf 'frr defaults datacenter\n' >> /etc/quagga/Quagga.conf

    printf 'username cumulus nopassword\n' >> /etc/quagga/Quagga.conf

    printf 'router-id 10.1.1.2\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65002\n  address-family l2vpn evpn\n  neighbor swp5 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65002\n  address-family l2vpn evpn\n  neighbor swp6 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65002\n  address-family l2vpn evpn\n  advertise-all-vni\n' >> /etc/quagga/Quagga.conf

    printf 'line vty\n' >> /etc/quagga/Quagga.conf

    cumulus@leaf02:mgmt-vrf:~$

     

     

    Leaf03 Configurations

     

     

     

    cumulus@leaf03:mgmt-vrf:~$ net show configuration commands

    net del all

    net add vxlan vni13 vxlan id 13

    net add vxlan vni13 vxlan local-tunnelip 10.1.1.3

    net add loopback lo ip address 10.1.1.3/32

    net add loopback lo alias loopback interface

    net add interface eth0 ip address dhcp

    net add interface eth0 alias management interface

    net add interface eth0 vrf mgmt

    net add interface swp5 alias to Spine01

    net add interface swp6 alias to Spine02

    net add interface swp7 alias to Server03

    net add interface swp7 link autoneg on

    net add interface swp7 bridge access 13

    net add bridge bridge vlan-aware

    net add bridge bridge ports vni13,vni24,swp7

    net add bridge bridge vids 13,24

    net add bridge bridge pvid 1

    net add vrf mgmt ip address 127.0.0.1/8

    net add vrf mgmt vrf-table auto

    net add vxlan vni13 bridge access 13

    net add vxlan vni13 bridge learning off

    net add vxlan vni13 stp bpduguard

    net add vxlan vni13 stp portbpdufilter

    net add hostname leaf03

    net add interface swp1-16 breakout 1x

    net add interface swp5-6 ipv6 nd ra-interval 10

    net del interface swp5-6 ipv6 nd suppress-ra

    net add routing service integrated-vtysh-config

    net add routing log timestamp precision 6

    net add bgp autonomous-system 65003

    net add bgp router-id 10.1.1.3

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor swp5 interface remote-as external

    net add bgp neighbor swp6 interface remote-as external

    net add bgp ipv4 unicast network 10.1.1.3/32

    net add bgp ipv4 unicast network 10.10.10.2/32

    net add time zone Etc/UTC

    net add time ntp server 0.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 1.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 2.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 3.cumulusnetworks.pool.ntp.org iburst

    net add time ntp source eth0

    net add dns nameserver ipv4 10.20.4.187 vrf mgmt

    net add snmp-server listening-address localhost

     

     

    There are some configuration commands that are not yet supported by nclu.

    The following will append those commands to the appropriate files.

    =======================================================================

    printf 'frr version 3.1+cl3u2\n' >> /etc/quagga/Quagga.conf

    printf 'frr defaults datacenter\n' >> /etc/quagga/Quagga.conf

    printf 'username cumulus nopassword\n' >> /etc/quagga/Quagga.conf

    printf 'router-id 10.1.1.3\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65003\n  address-family l2vpn evpn\n  neighbor swp5 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65003\n  address-family l2vpn evpn\n  neighbor swp6 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65003\n  address-family l2vpn evpn\n  advertise-all-vni\n' >> /etc/quagga/Quagga.conf

    printf 'line vty\n' >> /etc/quagga/Quagga.conf

    cumulus@leaf03:mgmt-vrf:~$

     

     

    Host01 Configurations

     

     

    root@mti-vishal-01:~# vi /etc/network/interfaces

    auto enp4s0f0

    iface enp4s0f0 inet dhcp

     

    auto enp129s0

    iface enp129s0 inet manual

        bond-master uplink

     

    auto enp3s0

    iface enp3s0 inet manual

        bond-master uplink

     

     

    auto uplink

    iface uplink inet static

      bond-slaves none

      bond-mode 802.3ad

      bond-miimon 100

      bond-lacp-rate 1

      bond-min-links 1

      bond-xmit-hash-policy layer3+4

      address 10.2.4.104

      netmask 255.255.255.0

     

    # install ifenslave

    # modprobe bonding

    ~

     

     

    auto enp4s0f0

    iface enp4s0f0 inet dhcp

     

    auto enp130s0

    iface enp130s0 inet static

        address 10.2.4.102

        netmask 255.255.255.0

     

     

    # install ifenslave

    Ping Verification

     

     

    modprobe bonding