EVPN Asymmetric Routing with Mellanox Switches

Version 1

    EVPN

     

    As explained RFC 7432, EVPN (stands for Ethernet Virtual Private Network) is new standard for making data center overlays which is becoming default standard on how the overlays are made in data centers, how data centers can be inter connected and how it can be done without any controller (which locks in customers with proeritary protocols).

    EVPN works as a control plane for VXLAN by distributing MAC and IP address for VTEPs (VXLAN Tunnel End Points). EVPN uses BGP to propagate the control plane information of MAC/IP. Because EVPN leverages the proven BGP protocol for control plane, it is becoming choice for making VXLANs.

    EVPN has different type of control packets (Type packets) which take care of making the control plane. Table 1.1 shows the details of such packets.

     

     

    Route Type

    Route Type Name

    Purpose

    Support on Mellanox Switches

    Type – 1

    Ethernet Auto-discovery Route

    • Needed for taking care of dual homes hosts
    • Auto Discovery per ESI for Fast Convergence & Spilt horizon
    • Auto Discovery per EVI route is used for Aliasing

    Need for Type – 1 route types

    Active - Active is taken care by MLAG

    EVPN interacts with MLAG

    Type – 2

    MAC/IP Route

    • Advertise host MAC and IP address as soon as hosts comes up.
    • Proactive learning suppress ARP (reducing ARP storms)

    Supported

    Type – 3

    Inclusive Multicast Route

    • Used to advertise the location of VTEPs in a VXLAN
    • Used to forward BUM traffic between VTEPs (Head end replication). This is used if the MAC is not known.

    Supported

    Type – 4

    Ethernet Segment

    • Needed for taking care of dual homed hosts
    • Route Discovery of nodes in redundancy group
    • Selection of Dedicated forwarder (DF-election)

    Need for Type – 4 route types

    Active - Active is taken care by MLAG

    EVPN interacts with MLAG

    Type – 5

    IP Prefix route

    • Advertise IP prefixes

    Supported

    Type – 1 and Type – 4 packets are used if the underlay is MPLS based. In case of IP fabric MLAG can take care of Type – 1 and Type – 4 scenarios.

     
     

     

    EVPN Routing

     

    RFC 7432 defines two modes of routing (routing is called IRB in EVPN terminology):

    1. Asymmetric routing: In this mode the routing happens at source VTEP (ingress) and on destination VTEP (egress) it is just bridging.

     

     

     

    2. Symmetric routing: In this mode the routing happens at source VTEP (ingress) as well as on destination VTEP (egress) along with bridging at both places.

     

    1.    

     

    This document goes in details of Asymmetric routing mode on Mellanox switches. The reference topology has been chosen to show both uplink modes (dual and single attached). On underlay BGP unnumbered is used. This document also includes detailed troubleshooting steps for Asymmetric configurations.

     

    Topology

     

    Configurations

     

     

    Spine01 Configurations

     

    cumulus@spine01:mgmt-vrf:~$ net show configuration commands

    net del all

    net add interface swp31-32 alias to Spine02

    net add loopback lo ip address 10.100.100.1/32

    net add loopback lo alias loopback interface

    net add interface eth0 ip address dhcp

    net add interface eth0 alias management interface

    net add interface eth0 vrf mgmt

    net add interface swp1 alias to Leaf01

    net add interface swp2 alias to Leaf02

    net add interface swp3 alias to Leaf03

    net add interface swp4 alias to Leaf04

    net add interface swp29 alias to Exit01

    net add interface swp30 alias to Exit02

    net add vrf mgmt ip address 127.0.0.1/8

    net add vrf mgmt vrf-table auto

    net add hostname spine01

    net add interface swp1-16 breakout 1x

    net add interface swp1-4,29-30 ipv6 nd ra-interval 10

    net del interface swp1-4,29-30 ipv6 nd suppress-ra

    net add routing service integrated-vtysh-config

    net add routing log file /var/log/quagga/quagga.log

    net add routing log timestamp precision 6

    net add bgp autonomous-system 65531

    net add bgp router-id 10.100.100.1

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor swp1 interface remote-as external

    net add bgp neighbor swp2 interface remote-as external

    net add bgp neighbor swp3 interface remote-as external

    net add bgp neighbor swp4 interface remote-as external

    net add bgp neighbor swp29 interface remote-as external

    net add bgp neighbor swp30 interface remote-as external

    net add bgp ipv4 unicast network 10.100.100.1/32

    net add time zone Etc/UTC

    net add time ntp server 0.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 1.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 2.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 3.cumulusnetworks.pool.ntp.org iburst

    net add time ntp source eth0

    net add dns nameserver ipv4 10.20.4.187 vrf mgmt

    net add snmp-server listening-address localhost

     

     

    There are some configuration commands that are not yet supported by nclu.

    The following will append those commands to the appropriate files.

    =======================================================================

    printf 'frr version 3.1+cl3u2\n' >> /etc/quagga/Quagga.conf

    printf 'frr defaults datacenter\n' >> /etc/quagga/Quagga.conf

    printf 'username cumulus nopassword\n' >> /etc/quagga/Quagga.conf

    printf 'router-id 10.100.100.1\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65531\n  address-family l2vpn evpn\n  neighbor swp1 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65531\n  address-family l2vpn evpn\n  neighbor swp2 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65531\n  address-family l2vpn evpn\n  neighbor swp3 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65531\n  address-family l2vpn evpn\n  neighbor swp4 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65531\n  address-family l2vpn evpn\n  neighbor swp29 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65531\n  address-family l2vpn evpn\n  neighbor swp30 activate\n' >> /etc/quagga/Quagga.conf

    printf 'line vty\n' >> /etc/quagga/Quagga.conf

    cumulus@spine01:mgmt-vrf:~$

     

     

    Spine02 Configurations

     

     

     

    cumulus@spine02:~$ net show configuration commands

    net del all

    net add interface swp31-32 alias to Spine02

    net add loopback lo ip address 10.100.100.2/32

    net add loopback lo alias loopback interface

    net add interface eth0 ip address dhcp

    net add interface eth0 alias management interface

    net add interface eth0 vrf mgmt

    net add interface swp1 alias to Leaf01

    net add interface swp2 alias to Leaf02

    net add interface swp3 alias to Leaf03

    net add interface swp4 alias to Leaf04

    net add interface swp29 alias to Exit01

    net add interface swp30 alias to Exit02

    net add vrf mgmt ip address 127.0.0.1/8

    net add vrf mgmt vrf-table auto

    net add hostname spine02

    net add interface swp1-16 breakout 1x

    net add interface swp1-4,29-30 ipv6 nd ra-interval 10

    net del interface swp1-4,29-30 ipv6 nd suppress-ra

    net add routing service integrated-vtysh-config

    net add routing log file /var/log/quagga/quagga.log

    net add routing log timestamp precision 6

    net add bgp autonomous-system 65532

    net add bgp router-id 10.100.100.2

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor swp1 interface remote-as external

    net add bgp neighbor swp2 interface remote-as external

    net add bgp neighbor swp3 interface remote-as external

    net add bgp neighbor swp4 interface remote-as external

    net add bgp neighbor swp29 interface remote-as external

    net add bgp neighbor swp30 interface remote-as external

    net add bgp ipv4 unicast network 10.100.100.2/32

    net add time zone Etc/UTC

    net add time ntp server 0.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 1.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 2.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 3.cumulusnetworks.pool.ntp.org iburst

    net add time ntp source eth0

    net add dns nameserver ipv4 10.20.4.187 vrf mgmt

    net add dns nameserver ipv4 10.20.4.167 vrf mgmt

    net add dns nameserver ipv4 10.2.8.150 vrf mgmt

    net add dns nameserver ipv4 10.2.8.3 vrf mgmt

    net add dns nameserver ipv4 10.2.8.4 vrf mgmt

    net add snmp-server listening-address localhost

     

     

    There are some configuration commands that are not yet supported by nclu.

    The following will append those commands to the appropriate files.

    =======================================================================

    printf 'frr version 3.1+cl3u2\n' >> /etc/quagga/Quagga.conf

    printf 'frr defaults datacenter\n' >> /etc/quagga/Quagga.conf

    printf 'username cumulus nopassword\n' >> /etc/quagga/Quagga.conf

    printf 'router-id 10.100.100.2\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65532\n  address-family l2vpn evpn\n  neighbor swp1 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65532\n  address-family l2vpn evpn\n  neighbor swp2 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65532\n  address-family l2vpn evpn\n  neighbor swp3 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65532\n  address-family l2vpn evpn\n  neighbor swp4 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65532\n  address-family l2vpn evpn\n  neighbor swp29 activate\n' >> /etc/quagga/Quagga.conf

    printf 'router bgp 65532\n  address-family l2vpn evpn\n  neighbor swp30 activate\n' >> /etc/quagga/Quagga.conf

    printf 'line vty\n' >> /etc/quagga/Quagga.conf

    cumulus@spine02:~$

     

     

     

    Leaf01 Configurations

     

     

     

     

    cumulus@leaf01:mgmt-vrf:~$ net show configuration commands

    net del all

    net add bond bond01 bond slaves swp7

    net add bond peerlink bond slaves swp15,swp16

    net add bond bond01 bridge access 13

    net add bond bond01 clag id 1

    net add bridge bridge ports bond01,peerlink,vni13,vni24

    net add bridge bridge pvid 1

    net add bridge bridge vids 13,24

    net add bridge bridge vlan-aware

    net add interface eth0 alias management interface

    net add interface eth0 ip address dhcp

    net add interface eth0 vrf mgmt

    net add interface peerlink.4094 clag backup-ip 10.1.1.2

    net add interface peerlink.4094 clag peer-ip 169.254.1.2

    net add interface peerlink.4094 clag priority 100

    net add interface peerlink.4094 clag sys-mac 44:39:39:ff:40:94

    net add interface peerlink.4094 ip address 169.254.1.1/30

    net add interface swp15-16 alias to Leaf02

    net add interface swp5 alias to Spine01

    net add interface swp6 alias to Spine02

    net add interface swp7 alias to Server01

    net add interface swp7 link autoneg on

    net add loopback lo alias loopback interface

    net add loopback lo clag vxlan-anycast-ip 10.10.10.1

    net add loopback lo ip address 10.1.1.1/32

    net add vlan 13 ip address 10.1.3.11/24

    net add vlan 13 ip address-virtual 00:00:00:00:00:13 10.1.3.1/24

    net add vlan 13 vlan-id 13

    net add vlan 13 vlan-raw-device bridge

    net add vlan 24 ip address 10.2.4.11/24

    net add vlan 24 ip address-virtual 00:00:00:00:00:24 10.2.4.1/24

    net add vlan 24 vlan-id 24

    net add vlan 24 vlan-raw-device bridge

    net add vrf mgmt ip address 127.0.0.1/8

    net add vrf mgmt vrf-table auto

    net add vxlan vni13 bridge access 13

    net add vxlan vni13 vxlan id 13

    net add vxlan vni13,24 bridge learning off

    net add vxlan vni13,24 stp bpduguard

    net add vxlan vni13,24 stp portbpdufilter

    net add vxlan vni13,24 vxlan local-tunnelip 10.1.1.1

    net add vxlan vni24 bridge access 24

    net add vxlan vni24 vxlan id 24

    net add hostname leaf01

    net add interface swp1-16 breakout 1x

    net add interface swp5-6 ipv6 nd ra-interval 10

    net del interface swp5-6 ipv6 nd suppress-ra

    net add routing defaults datacenter

    net add routing service integrated-vtysh-config

    net add routing log timestamp precision 6

    net add bgp autonomous-system 65001

    net add bgp router-id 10.1.1.1

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor swp5 interface remote-as external

    net add bgp neighbor swp6 interface remote-as external

    net add bgp ipv4 unicast network 10.1.1.1/32

    net add bgp ipv4 unicast network 10.10.10.1/32

    net add bgp l2vpn evpn  neighbor swp5 activate

    net add bgp l2vpn evpn  neighbor swp6 activate

    net add bgp l2vpn evpn  advertise-all-vni

    net add time zone Etc/UTC

    net add time ntp server 0.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 1.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 2.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 3.cumulusnetworks.pool.ntp.org iburst

    net add time ntp source eth0

    net add dns nameserver ipv4 10.20.4.187 vrf mgmt

    net add dns nameserver ipv4 10.20.4.167 vrf mgmt

    net add dns nameserver ipv4 10.2.8.150 vrf mgmt

    net add dns nameserver ipv4 10.2.8.3 vrf mgmt

    net add dns nameserver ipv4 10.2.8.4 vrf mgmt

    net add snmp-server listening-address localhost

     

     

    There are some configuration commands that are not yet supported by nclu.

    The following will append those commands to the appropriate files.

    =======================================================================

    printf 'username cumulus nopassword\n' >> /etc/quagga/Quagga.conf

    printf 'router-id 10.1.1.1\n' >> /etc/quagga/Quagga.conf

    cumulus@leaf01:mgmt-vrf:~$

     

     

     

    Leaf02 Configurations

     

     

     

     

    cumulus@leaf02:mgmt-vrf:~$ net show configuration commands

    net del all

    net add bond bond01 bond slaves swp7

    net add bond peerlink bond slaves swp1,swp2

    net add bond bond01 bridge access 13

    net add bond bond01 clag id 1

    net add bridge bridge ports bond01,peerlink,vni13,vni24

    net add bridge bridge pvid 1

    net add bridge bridge vids 13,24

    net add bridge bridge vlan-aware

    net add interface eth0 alias management interface

    net add interface eth0 ip address dhcp

    net add interface eth0 vrf mgmt

    net add interface peerlink.4094 clag backup-ip 10.1.1.1

    net add interface peerlink.4094 clag peer-ip 169.254.1.1

    net add interface peerlink.4094 clag priority 200

    net add interface peerlink.4094 clag sys-mac 44:39:39:ff:40:94

    net add interface peerlink.4094 ip address 169.254.1.2/30

    net add interface swp1-2 alias to Leaf01

    net add interface swp5 alias to Spine01

    net add interface swp6 alias to Spine02

    net add interface swp7 alias to Server01

    net add interface swp7 link autoneg on

    net add loopback lo alias loopback interface

    net add loopback lo clag vxlan-anycast-ip 10.10.10.1

    net add loopback lo ip address 10.1.1.2/32

    net add vlan 13 ip address 10.1.3.12/24

    net add vlan 13 ip address-virtual 00:00:00:00:00:13 10.1.3.1/24

    net add vlan 13 vlan-id 13

    net add vlan 13 vlan-raw-device bridge

    net add vlan 24 ip address 10.2.4.12/24

    net add vlan 24 ip address-virtual 00:00:00:00:00:24 10.2.4.1/24

    net add vlan 24 vlan-id 24

    net add vlan 24 vlan-raw-device bridge

    net add vrf mgmt ip address 127.0.0.1/8

    net add vrf mgmt vrf-table auto

    net add vxlan vni13 bridge access 13

    net add vxlan vni13 vxlan id 13

    net add vxlan vni13,24 bridge learning off

    net add vxlan vni13,24 stp bpduguard

    net add vxlan vni13,24 stp portbpdufilter

    net add vxlan vni13,24 vxlan local-tunnelip 10.1.1.2

    net add vxlan vni24 bridge access 24

    net add vxlan vni24 vxlan id 24

    net add hostname leaf02

    net add interface swp1-32 breakout 1x

    net add interface swp5-6 ipv6 nd ra-interval 10

    net del interface swp5-6 ipv6 nd suppress-ra

    net add routing defaults datacenter

    net add routing service integrated-vtysh-config

    net add routing log file /var/log/quagga/quagga.log

    net add routing log timestamp precision 6

    net add bgp autonomous-system 65002

    net add bgp router-id 10.1.1.2

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor swp5 interface remote-as external

    net add bgp neighbor swp6 interface remote-as external

    net add bgp ipv4 unicast network 10.1.1.2/32

    net add bgp ipv4 unicast network 10.10.10.1/32

    net add bgp l2vpn evpn  neighbor swp5 activate

    net add bgp l2vpn evpn  neighbor swp6 activate

    net add bgp l2vpn evpn  advertise-all-vni

    net add time zone Etc/UTC

    net add time ntp server 0.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 1.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 2.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 3.cumulusnetworks.pool.ntp.org iburst

    net add time ntp source eth0

    net add dns nameserver ipv4 10.20.4.187 vrf mgmt

    net add dns nameserver ipv4 10.20.4.167 vrf mgmt

    net add dns nameserver ipv4 10.2.8.150 vrf mgmt

    net add dns nameserver ipv4 10.2.8.3 vrf mgmt

    net add dns nameserver ipv4 10.2.8.4 vrf mgmt

    net add snmp-server listening-address localhost

     

     

    There are some configuration commands that are not yet supported by nclu.

    The following will append those commands to the appropriate files.

    =======================================================================

    printf 'username cumulus nopassword\n' >> /etc/quagga/Quagga.conf

    printf 'router-id 10.1.1.2\n' >> /etc/quagga/Quagga.conf

    cumulus@leaf02:mgmt-vrf:~$

     

     

    Leaf03 Configurations

     

     

     

     

    cumulus@leaf03:mgmt-vrf:~$ net show configuration commands

    net del all

    net add bridge bridge ports vni13,vni24,swp7

    net add bridge bridge pvid 1

    net add bridge bridge vids 13,24

    net add bridge bridge vlan-aware

    net add interface eth0 alias management interface

    net add interface eth0 ip address dhcp

    net add interface eth0 vrf mgmt

    net add interface swp5 alias to Spine01

    net add interface swp6 alias to Spine02

    net add interface swp7 alias to Server

    net add interface swp7 bridge access 24

    net add interface swp7 link autoneg on

    net add loopback lo alias loopback interface

    net add loopback lo ip address 10.1.1.3/32

    net add vlan 13 ip address 10.1.3.13/24

    net add vlan 13 ip address-virtual 00:00:00:00:00:13 10.1.3.1/24

    net add vlan 13 vlan-id 13

    net add vlan 13 vlan-raw-device bridge

    net add vlan 24 ip address 10.2.4.13/24

    net add vlan 24 ip address-virtual 00:00:00:00:00:24 10.2.4.1/24

    net add vlan 24 vlan-id 24

    net add vlan 24 vlan-raw-device bridge

    net add vrf mgmt ip address 127.0.0.1/8

    net add vrf mgmt vrf-table auto

    net add vxlan vni13 bridge access 13

    net add vxlan vni13 vxlan id 13

    net add vxlan vni13,24 bridge learning off

    net add vxlan vni13,24 stp bpduguard

    net add vxlan vni13,24 stp portbpdufilter

    net add vxlan vni13,24 vxlan local-tunnelip 10.1.1.3

    net add vxlan vni24 bridge access 24

    net add vxlan vni24 vxlan id 24

    net add hostname leaf03

    net add interface swp1-16 breakout 1x

    net add interface swp5-6 ipv6 nd ra-interval 10

    net del interface swp5-6 ipv6 nd suppress-ra

    net add routing defaults datacenter

    net add routing service integrated-vtysh-config

    net add routing log timestamp precision 6

    net add bgp autonomous-system 65003

    net add bgp router-id 10.1.1.3

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor swp5 interface remote-as external

    net add bgp neighbor swp6 interface remote-as external

    net add bgp ipv4 unicast network 10.1.1.3/32

    net add bgp ipv4 unicast network 10.10.10.2/32

    net add bgp l2vpn evpn  neighbor swp5 activate

    net add bgp l2vpn evpn  neighbor swp6 activate

    net add bgp l2vpn evpn  advertise-all-vni

    net add time zone Etc/UTC

    net add time ntp server 0.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 1.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 2.cumulusnetworks.pool.ntp.org iburst

    net add time ntp server 3.cumulusnetworks.pool.ntp.org iburst

    net add time ntp source eth0

    net add dns nameserver ipv4 10.20.4.187 vrf mgmt

    net add dns nameserver ipv4 10.20.4.167 vrf mgmt

    net add dns nameserver ipv4 10.2.8.150 vrf mgmt

    net add dns nameserver ipv4 10.2.8.3 vrf mgmt

    net add dns nameserver ipv4 10.2.8.4 vrf mgmt

    net add snmp-server listening-address localhost

     

     

    There are some configuration commands that are not yet supported by nclu.

    The following will append those commands to the appropriate files.

    =======================================================================

    printf 'username cumulus nopassword\n' >> /etc/quagga/Quagga.conf

    printf 'router-id 10.1.1.3\n' >> /etc/quagga/Quagga.conf

    cumulus@leaf03:mgmt-vrf:~$

     

     

     

     

    Host01 Configurations

     

     

    auto enp4s0f0

    iface enp4s0f0 inet dhcp

     

    auto enp130s0

    iface enp130s0 inet static

        address 10.2.4.102

        netmask 255.255.255.0

        post-up ip route add 10.1.3.0/24 via 10.2.4.1 dev enp130s0

     

     

    # install ifenslave

    # modprobe bonding

     

    ~

     

     

    auto enp4s0f0

    iface enp4s0f0 inet dhcp

     

    auto enp129s0

    iface enp129s0 inet manual

        bond-master uplink

     

    auto enp3s0

    iface enp3s0 inet manual

        bond-master uplink

     

     

    auto uplink

    iface uplink inet static

      bond-slaves none

      bond-mode 802.3ad

      bond-miimon 100

      bond-lacp-rate 1

      bond-min-links 1

      bond-xmit-hash-policy layer3+4

      address 10.1.3.103

      netmask 255.255.255.0

     

    post-up ip route add 10.2.4.0/24 via 10.1.3.1

     

    # install ifenslave

    # modprobe bonding

     

    Ping Verification

     

     

    root@mti-vishal-02:~# ifconfig

    enp130s0  Link encap:Ethernet  HWaddr 00:02:c9:32:6e:20

              inet addr:10.2.4.102  Bcast:0.0.0.0  Mask:255.255.255.0

              inet6 addr: fe80::202:c9ff:fe32:6e20/64 Scope:Link

              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

              RX packets:11225 errors:0 dropped:0 overruns:0 frame:0

              TX packets:9790 errors:0 dropped:0 overruns:0 carrier:0

              collisions:0 txqueuelen:1000

              RX bytes:1040566 (1.0 MB)  TX bytes:947186 (947.1 KB)

     

    enp4s0f0  Link encap:Ethernet  HWaddr 00:25:90:94:91:20

              inet addr:10.20.1.81  Bcast:10.20.255.255  Mask:255.255.0.0

              inet6 addr: fe80::225:90ff:fe94:9120/64 Scope:Link

              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

              RX packets:2083778 errors:0 dropped:0 overruns:0 frame:0

              TX packets:28754 errors:0 dropped:0 overruns:0 carrier:0

              collisions:0 txqueuelen:1000

              RX bytes:258469510 (258.4 MB)  TX bytes:2971944 (2.9 MB)

              Memory:dfc60000-dfc7ffff

     

    lo        Link encap:Local Loopback

              inet addr:127.0.0.1  Mask:255.0.0.0

              inet6 addr: ::1/128 Scope:Host

              UP LOOPBACK RUNNING  MTU:65536  Metric:1

              RX packets:33805 errors:0 dropped:0 overruns:0 frame:0

              TX packets:33805 errors:0 dropped:0 overruns:0 carrier:0

              collisions:0 txqueuelen:1

              RX bytes:2152756 (2.1 MB)  TX bytes:2152756 (2.1 MB)

     

    root@mti-vishal-02:~# ping 10.1.3.103

    PING 10.1.3.103 (10.1.3.103) 56(84) bytes of data.

    64 bytes from 10.1.3.103: icmp_seq=1 ttl=63 time=0.143 ms

    64 bytes from 10.1.3.103: icmp_seq=2 ttl=63 time=0.153 ms

    64 bytes from 10.1.3.103: icmp_seq=3 ttl=63 time=0.128 ms

    64 bytes from 10.1.3.103: icmp_seq=4 ttl=63 time=0.141 ms

    ^C

    --- 10.1.3.103 ping statistics ---

    4 packets transmitted, 4 received, 0% packet loss, time 2999ms

    rtt min/avg/max/mdev = 0.128/0.141/0.153/0.012 ms

    root@mti-vishal-02:~#

     

     

    Troubleshooting from Leaf – 1 perspective (refer topology for IPs and MACs)

    -          Step – 1: Check for BGP routes

     

    cumulus@leaf01:mgmt-vrf:/etc/frr$ net show route

     

    show ip route

    =============

    Codes: K - kernel route, C - connected, S - static, R - RIP,

           O - OSPF, I - IS-IS, B - BGP, P - PIM, E - EIGRP, N - NHRP,

           T - Table, v - VNC, V - VNC-Direct, A - Babel,

           > - selected route, * - FIB route

     

    C>* 10.1.1.1/32 is directly connected, lo, 00:03:38

    B>* 10.1.1.2/32 [20/0] via fe80::7efe:90ff:fefc:7adc, swp5, 00:03:35

      *                    via fe80::7efe:90ff:fefc:7c5c, swp6, 00:03:35

    B>* 10.1.1.3/32 [20/0] via fe80::7efe:90ff:fefc:7adc, swp5, 00:03:35

      *                    via fe80::7efe:90ff:fefc:7c5c, swp6, 00:03:35

    C * 10.1.3.0/24 is directly connected, vlan13-v0, 00:03:38

    C>* 10.1.3.0/24 is directly connected, vlan13, 00:03:38

    C * 10.2.4.0/24 is directly connected, vlan24-v0, 00:03:38

    C>* 10.2.4.0/24 is directly connected, vlan24, 00:03:38

    C>* 10.10.10.1/32 is directly connected, lo, 00:03:38

    B>* 10.100.100.1/32 [20/0] via fe80::7efe:90ff:fefc:7adc, swp5, 00:03:36

    B>* 10.100.100.2/32 [20/0] via fe80::7efe:90ff:fefc:7c5c, swp6, 00:03:36

    C>* 169.254.1.0/30 is directly connected, peerlink.4094, 00:03:38

     

     

    show ipv6 route

    ===============

    Codes: K - kernel route, C - connected, S - static, R - RIPng,

           O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,

           v - VNC, V - VNC-Direct, A - Babel,

           > - selected route, * - FIB route

     

    C * fe80::/64 is directly connected, swid0_eth, 00:03:38

    C * fe80::/64 is directly connected, peerlink.4094, 00:03:38

    C * fe80::/64 is directly connected, vlan24-v0, 00:03:38

    C * fe80::/64 is directly connected, vlan24, 00:03:38

    C * fe80::/64 is directly connected, vlan13-v0, 00:03:38

    C * fe80::/64 is directly connected, vlan13, 00:03:38

    C * fe80::/64 is directly connected, bridge, 00:03:38

    C * fe80::/64 is directly connected, swp6, 00:03:38

    C>* fe80::/64 is directly connected, swp5, 00:03:38

    cumulus@leaf01:mgmt-vrf:/etc/frr$

     

    -          Step – 2: Check for EVPN Control plane (EVPN Routes – MAC/IP)

     

    cumulus@leaf01:mgmt-vrf:/etc/frr$ net show bgp evpn route

    BGP table version is 3, local router ID is 10.1.1.1

    Status codes: s suppressed, d damped, h history, * valid, > best, i - internal

    Origin codes: i - IGP, e - EGP, ? - incomplete

    EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]

    EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]

     

       Network          Next Hop            Metric LocPrf Weight Path

    Route Distinguisher: 10.1.1.1:2

    *> [2]:[0]:[0]:[48]:[7c:fe:90:81:48:a0]

                        10.10.10.1                         32768 i

    *> [2]:[0]:[0]:[48]:[7c:fe:90:81:48:a0]:[32]:[10.1.3.103]

                        10.10.10.1                         32768 i

    *> [3]:[0]:[32]:[10.10.10.1]

                        10.10.10.1                         32768 i

    Route Distinguisher: 10.1.1.1:3

    *> [3]:[0]:[32]:[10.10.10.1]

                        10.10.10.1                         32768 i

    Route Distinguisher: 10.1.1.3:2

    *  [3]:[0]:[32]:[10.1.1.3]

                        10.1.1.3                               0 65532 65003 i

    *> [3]:[0]:[32]:[10.1.1.3]

                        10.1.1.3                               0 65531 65003 i

    Route Distinguisher: 10.1.1.3:3

    *  [2]:[0]:[0]:[48]:[00:02:c9:32:6e:20]

                        10.1.1.3                               0 65532 65003 i

    *> [2]:[0]:[0]:[48]:[00:02:c9:32:6e:20]

                        10.1.1.3                               0 65531 65003 i

    *  [2]:[0]:[0]:[48]:[00:02:c9:32:6e:20]:[32]:[10.2.4.102]

                        10.1.1.3                               0 65532 65003 i

    *> [2]:[0]:[0]:[48]:[00:02:c9:32:6e:20]:[32]:[10.2.4.102]

                        10.1.1.3                               0 65531 65003 i

    *  [3]:[0]:[32]:[10.1.1.3]

                        10.1.1.3                               0 65532 65003 i

    *> [3]:[0]:[32]:[10.1.1.3]

                        10.1.1.3                               0 65531 65003 i

     

    Displayed 8 prefixes (12 paths)

    cumulus@leaf01:mgmt-vrf:/etc/frr$

     

    -          Step – 3: Check for learned VNIs

     

    cumulus@leaf01:mgmt-vrf:/etc/frr$ net show evpn vni

    VNI        Type VxLAN IF              # MACs   # ARPs   # Remote VTEPs  Tenant VRF

    24         L2   vni24                 1        5        1               Default-IP-Routing-Table

    13         L2   vni13                 1        5        1               Default-IP-Routing-Table

    cumulus@leaf01:mgmt-vrf:/etc/frr$

     

    -          Step – 4: Check for learned MAC and remote VTEP IPs

     

    cumulus@leaf01:mgmt-vrf:/etc/frr$ net show evpn mac vni all

     

    VNI 24 #MACs (local and remote) 1

     

    MAC               Type   Intf/Remote VTEP      VLAN

    24:8a:07:55:1d:80 local  vlan24                24

    00:00:00:00:00:24 local  vlan24-v0             24

    00:02:c9:32:6e:20 remote 10.1.1.3

     

    VNI 13 #MACs (local and remote) 1

     

    MAC               Type   Intf/Remote VTEP      VLAN

    24:8a:07:55:1d:80 local  vlan13                13

    7c:fe:90:81:48:a0 local  bond01                13

    00:00:00:00:00:13 local  vlan13-v0             13

    cumulus@leaf01:mgmt-vrf:/etc/frr$

     

     

     

     

    1.      Symmetric routing: In this mode the routing happens at source VTEP (ingress) as well as on destination VTEP (egress) along with bridging at both places.