EVPN Asymmetric Routing with Type5 on Border leaf

Version 1

    EVPN Asymmetric Routing with Type5 on Border leaf.

     

    This article will explain how to deploy EVPN Asymmetric routing with Type5 prefix-routes advertised from the Border leaf, by using EVPN Type5 routes we will be able to connect our EVPN/VXLAN fabric to networks located out of our VXLAN domain.

     

    Topology:

     

     

     

     

     

     

    *Cumulus Linux uses the L3 VNI in Asymmetric vxlan to advertise only the type5 prefix routes.

     

    We will use VNI4001 as our L3 VNI to allow Type5 advertisement.

     

     

    Configuration

     

     

    TOR1

     

     

    net add interface swp2 ipv6 nd ra-interval 10

    net del interface swp2 ipv6 nd suppress-ra

    net add vrf vrf1 vni 104001 prefix-routes-only

    net add bgp autonomous-system 65003

    net add bgp router-id 10.0.0.3

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor FABRIC peer-group

    net add bgp neighbor FABRIC remote-as external

    net add bgp neighbor FABRIC capability extended-nexthop

    net add bgp neighbor swp2 interface peer-group FABRIC

    net add bgp ipv4 unicast network 10.0.0.3/32

    net add bgp ipv6 unicast neighbor FABRIC activate

    net add bgp l2vpn evpn neighbor FABRIC activate

    net add bgp l2vpn evpn advertise-all-vni

    net add bgp l2vpn evpn advertise ipv4 unicast

    net add interface swp1-16 breakout 1x

    net add vxlan vtep10 vxlan id 10010

    net add vxlan vtep20 vxlan id 10020

    net add vxlan vxlan4001 vxlan id 104001

    net add bridge bridge ports swp1,vtep10,vtep20,vxlan4001

    net add bridge bridge vids 10,20,4001

    net add bridge bridge vlan-aware

    net add interface swp1 bridge access 10

    net add interface swp1-2 mtu 9216

    net add loopback lo ip address 10.0.0.3/32

    net add vlan 10 ip address 192.168.10.101/24

    net add vlan 10 ip address-virtual 00:00:00:00:00:1a 192.168.10.254/24

    net add vlan 10 vlan-id 10

    net add vlan 10 vlan-raw-device bridge

    net add vlan 10 vrf vrf1

    net add vlan 20 ip address 192.168.20.101/24

    net add vlan 20 ip address-virtual 00:00:00:00:00:2a 192.168.20.254/24

    net add vlan 20 vlan-id 20

    net add vlan 20 vlan-raw-device bridge

    net add vlan 20 vrf vrf1

    net add vlan 4001 vlan-id 4001

    net add vlan 4001 vlan-raw-device bridge

    net add vlan 4001 vrf vrf1

    net add vrf vrf1 vrf-table auto

    net add vxlan vtep10 bridge access 10

    net add vxlan vtep10,20,vxlan4001 bridge arp-nd-suppress on

    net add vxlan vtep10,20,vxlan4001 bridge learning off

    net add vxlan vtep10,20,vxlan4001 mtu 9216

    net add vxlan vtep10,20,vxlan4001 stp bpduguard

    net add vxlan vtep10,20,vxlan4001 stp portbpdufilter

    net add vxlan vtep10,20,vxlan4001 vxlan local-tunnelip 10.0.0.3

    net add vxlan vtep20 bridge access 20

    net add vxlan vxlan4001 bridge access 4001

     

     
     

    TOR2

     

     

    net add interface swp2 ipv6 nd ra-interval 10

    net del interface swp2 ipv6 nd suppress-ra

    net add vrf vrf1 vni 104001 prefix-routes-only

    net add bgp autonomous-system 65004

    net add bgp router-id 10.0.0.4

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor FABRIC peer-group

    net add bgp neighbor FABRIC remote-as external

    net add bgp neighbor FABRIC capability extended-nexthop

    net add bgp neighbor swp2 interface peer-group FABRIC

    net add bgp ipv4 unicast network 10.0.0.4/32

    net add bgp ipv6 unicast neighbor FABRIC activate

    net add bgp l2vpn evpn  neighbor FABRIC activate

    net add bgp l2vpn evpn  advertise-all-vni

    net add bgp l2vpn evpn  advertise ipv4 unicast

    net add interface swp1-16 breakout 1x

    net add vxlan vtep10 vxlan id 10010

    net add vxlan vtep20 vxlan id 10020

    net add vxlan vxlan4001 vxlan id 104001

    net add bridge bridge ports swp1,vtep10,vtep20,vxlan4001

    net add bridge bridge vids 10,20,4001

    net add bridge bridge vlan-aware

    net add interface swp1 bridge access 10

    net add interface swp1-2 mtu 9216

    net add loopback lo ip address 10.0.0.3/32

    net add vlan 10 ip address 192.168.10.101/24

    net add vlan 10 ip address-virtual 00:00:00:00:00:1a 192.168.10.254/24

    net add vlan 10 vlan-id 10

    net add vlan 10 vlan-raw-device bridge

    net add vlan 10 vrf vrf1

    net add vlan 20 ip address 192.168.20.101/24

    net add vlan 20 ip address-virtual 00:00:00:00:00:2a 192.168.20.254/24

    net add vlan 20 vlan-id 20

    net add vlan 20 vlan-raw-device bridge

    net add vlan 20 vrf vrf1

    net add vlan 4001 vlan-id 4001

    net add vlan 4001 vlan-raw-device bridge

    net add vlan 4001 vrf vrf1

    net add vrf vrf1 vrf-table auto

    net add vxlan vtep10 bridge access 10

    net add vxlan vtep10,20,vxlan4001 bridge arp-nd-suppress on

    net add vxlan vtep10,20,vxlan4001 bridge learning off

    net add vxlan vtep10,20,vxlan4001 mtu 9216

    net add vxlan vtep10,20,vxlan4001 stp bpduguard

    net add vxlan vtep10,20,vxlan4001 stp portbpdufilter

    net add vxlan vtep10,20,vxlan4001 vxlan local-tunnelip 10.0.0.4

    net add vxlan vtep20 bridge access 20

    net add vxlan vxlan4001 bridge access 4001

       

     

    SPINE

     

     

    net add interface swp1-3 ipv6 nd ra-interval 10

    net del interface swp1-3 ipv6 nd suppress-ra

    net add bgp autonomous-system 65100

    net add bgp router-id 10.0.0.100

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor FABRIC peer-group

    net add bgp neighbor FABRIC remote-as external

    net add bgp neighbor FABRIC capability extended-nexthop

    net add bgp neighbor swp1 interface peer-group FABRIC

    net add bgp neighbor swp2 interface peer-group FABRIC

    net add bgp neighbor swp3 interface peer-group FABRIC

    net add bgp ipv6 unicast neighbor FABRIC activate

    net add bgp l2vpn evpn  neighbor FABRIC activate

    net add bgp l2vpn evpn  advertise-all-vni

    net add bgp l2vpn evpn  advertise ipv4 unicast

    net add interface eth0 ip address dhcp

    net add interface swp1-3 mtu 9216

     

    Border leaf

     

    The border leaf is connected to the router via OSPF, these we will need to redistribute routes from OSPF into the EVPN Overlay network.

     

    net add interface swp50 ipv6 nd ra-interval 10

    net del interface swp50 ipv6 nd suppress-ra

    net add vrf vrf1 vni 104001 prefix-routes-only

    net add bgp autonomous-system 65004

    net add bgp router-id 10.0.0.20

    net add bgp bestpath as-path multipath-relax

    net add bgp neighbor FABRIC peer-group

    net add bgp neighbor FABRIC remote-as external

    net add bgp neighbor FABRIC capability extended-nexthop

    net add bgp neighbor swp50 interface peer-group FABRIC

    net add bgp ipv4 unicast network 10.0.0.20/32

    net add bgp ipv4 unicast redistribute static

    net add bgp ipv6 unicast neighbor FABRIC activate

    net add bgp l2vpn evpn  neighbor FABRIC activate

    net add bgp l2vpn evpn  advertise-all-vni

    net add bgp l2vpn evpn  advertise ipv4 unicast

    net add bgp vrf vrf1 autonomous-system 65004

    net add bgp vrf vrf1 router-id 10.0.0.20

    net add bgp vrf vrf1 ipv4 unicast network 45.45.45.0/24

    net add bgp vrf vrf1 ipv4 unicast redistribute ospf

    net add bgp vrf vrf1 l2vpn evpn  advertise ipv4 unicast

    net add ospf vrf vrf1

    net add ospf vrf vrf1 redistribute connected

    net add ospf vrf vrf1 network 45.45.45.0/24 area 0

    net add vxlan vtep10 vxlan id 10010

    net add vxlan vtep20 vxlan id 10020

    net add vxlan vxlan4001 vxlan id 104001

    net add bridge bridge ports vtep10,vtep20,vxlan4001

    net add bridge bridge vids 10,20,4001

    net add bridge bridge vlan-aware

    net add bridge stp off

    net add interface swp1-48,51-56

    net add interface swp49 ip address 45.45.45.2/30

    net add interface swp49 vrf vrf1

    net add interface swp49-50 mtu 9216

    net add loopback lo ip address 10.0.0.20/32

    net add vlan 10 ip address 192.168.10.101/24

    net add vlan 10 ip address-virtual 00:00:00:00:00:1a 192.168.10.254/24

    net add vlan 10 vlan-id 10

    net add vlan 10 vlan-raw-device bridge

    net add vlan 10 vrf vrf1

    net add vlan 20 ip address 192.168.20.101/24

    net add vlan 20 ip address-virtual 00:00:00:00:00:2a 192.168.20.254/24

    net add vlan 20 vlan-id 20

    net add vlan 20 vlan-raw-device bridge

    net add vlan 20 vrf vrf1

    net add vlan 4001 vlan-id 4001

    net add vlan 4001 vlan-raw-device bridge

    net add vlan 4001 vrf vrf1

    net add vrf vrf1 vrf-table auto

    net add vxlan vtep10 bridge access 10

    net add vxlan vtep10,20,vxlan4001 bridge arp-nd-suppress on

    net add vxlan vtep10,20,vxlan4001 bridge learning off

    net add vxlan vtep10,20,vxlan4001 mtu 9216

    net add vxlan vtep10,20,vxlan4001 stp bpduguard

    net add vxlan vtep10,20,vxlan4001 stp portbpdufilter

    net add vxlan vtep10,20,vxlan4001 vxlan local-tunnelip 10.0.0.20

    net add vxlan vtep20 bridge access 20

    net add vxlan vxlan4001 bridge access 4001

     

    Controlling Which RIB Routes Are Injected into EVPN

     

    By default, when announcing IP prefixes in the BGP RIB as EVPN type-5 routes, all routes in the BGP RIB are picked for advertisement as EVPN type-5 routes. You can use a route map to allow selective advertisement of routes from the BGP RIB as EVPN type-5 routes.

     

    The following command binds a route map filter to IPv4 EVPN type-5 route advertisement:

     

    net add bgp vrf vrf1 l2vpn evpn advertise ipv4 unicast route-map map1

     

     

    Router

     

     

    net add interface swp1-2 ospf area 0

    net add ospf vrf vrf1

    net add ospf vrf vrf1 redistribute bgp

    net add ospf vrf vrf1 network 45.45.45.0/30 area 0

    net add ospf vrf vrf1 network 192.168.168.0/24 area 0

    net add interface swp1 ip address 45.45.45.1/30

    net add interface swp1-2 mtu 9216

    net add interface swp2 ip address 192.168.168.254/24

    net add interface swp3-16

     

     
     

    Validation

     

    Validate that the Router is seeing our vxlan fabric subnets:

     

     

    cumulus@Router:~$ net show route

     

    show ip route

    =============

    Codes: K - kernel route, C - connected, S - static, R - RIP,

           O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,

           T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,

           F - PBR,

           > - selected route, * - FIB route

     

    K>* 0.0.0.0/0 [0/0] via 10.7.156.1, eth0, 01w6d03h

    C>* 10.7.156.0/22 is directly connected, eth0, 01w6d03h

    O   45.45.45.0/30 [110/1] is directly connected, swp1, 01w4d00h

    C>* 45.45.45.0/30 is directly connected, swp1, 01w4d00h

    O>* 192.168.10.0/24 [110/20] via 45.45.45.2, swp1, 01w0d02h

    O>* 192.168.20.0/24 [110/20] via 45.45.45.2, swp1, 01w0d02h

    O   192.168.168.0/24 [110/1] is directly connected, swp2, 01w4d21h

    C>* 192.168.168.0/24 is directly connected, swp2, 01w4d22h

     

    Validate OSPF route is seen on the Border leaf:

     

     

    cumulus@BorderLeaf:~$ net show route vrf vrf1

     

    show ip route vrf vrf1

    =======================

    Codes: K - kernel route, C - connected, S - static, R - RIP,

           O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,

           T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,

           F - PBR,

           > - selected route, * - FIB route

     

     

    VRF vrf1:

    K * 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 01w3d21h

    O   45.45.45.0/30 [110/1] is directly connected, swp49, 01w0d00h

    C>* 45.45.45.0/30 is directly connected, swp49, 01w0d00h

    C * 192.168.10.0/24 is directly connected, vlan10-v0, 00:10:35

    C>* 192.168.10.0/24 is directly connected, vlan10, 6d23h54m

    C * 192.168.20.0/24 is directly connected, vlan20-v0, 00:10:35

    C>* 192.168.20.0/24 is directly connected, vlan20, 6d23h54m

    O>* 192.168.168.0/24 [110/2] via 45.45.45.1, swp49, 01w0d00h

     

    Validate that the OSPF route is advertised into BGP on the Border leaf:

     

     

    cumulus@BorderLeaf:~$ net show bgp vrf vrf1

     

    show bgp vrf vrf1 ipv4 unicast

    ==============================

    BGP table version is 1, local router ID is 10.0.0.20

    Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,

                  i internal, r RIB-failure, S Stale, R Removed

    Origin codes: i - IGP, e - EGP, ? - incomplete

     

       Network          Next Hop            Metric LocPrf Weight Path

       45.45.45.0/24    0.0.0.0                  0         32768 i

    *> 192.168.168.0 45.45.45.1               2         32768 ?

     

    Validate that the route is seen inside the EVPN on the Border leaf:

     

     

    cumulus@Border:~$ net show bgp evpn route type prefix

    BGP table version is 5, local router ID is 10.0.0.20

    Status codes: s suppressed, d damped, h history, * valid, > best, i - internal

    Origin codes: i - IGP, e - EGP, ? - incomplete

    EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]

    EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]

    EVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]

     

       Network          Next Hop            Metric LocPrf Weight Path

    Route Distinguisher: 10.0.0.20:2

    *> [5]:[0]:[0]:[24]:[192.168.168.0]

    10.0.0.20                2         32768 ?

     

     

    Validate that the advertised prefix is seen by the TOR via EVPN:

     

     

    cumulus@TOR1:~$ net show bgp evpn route type prefix

    BGP table version is 5, local router ID is 10.0.0.3

    Status codes: s suppressed, d damped, h history, * valid, > best, i - internal

    Origin codes: i - IGP, e - EGP, ? - incomplete

    EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]

    EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]

    EVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]

     

       Network          Next Hop            Metric LocPrf Weight Path

    Route Distinguisher: 10.0.0.20:2

    *> [5]:[0]:[0]:[24]:[192.168.168.0]

    10.0.0.20                              0 65100 65004 ?

     

    Validate that the advertised prefix is set to the TOR routing table:

     

     

    cumulus@TOR1:~$ net show route vrf vrf1

     

    show ip route vrf vrf1

    =======================

    Codes: K - kernel route, C - connected, S - static, R - RIP,

           O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,

           T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,

           F - PBR,

           > - selected route, * - FIB route

     

     

    VRF vrf1:

    K * 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 01w0d00h

    C * 192.168.10.0/24 is directly connected, vlan10-v0, 01:14:15

    C>* 192.168.10.0/24 is directly connected, vlan10, 01w0d00h

    C * 192.168.20.0/24 is directly connected, vlan20-v0, 01:14:15

    C>* 192.168.20.0/24 is directly connected, vlan20, 01w0d00h

    B>* 192.168.168.0/24 [20/0] via 10.0.0.20, vlan4001 onlink, 6d19h41m

     

     

     

     

     

    Let’s run some traffic between 192.168.10.1(Server on TOR1) and 192.168.168.1(Server connected to the Router) :

     

     

     

     

    [root@192.168.168.1~]# iperf3 -c 192.168.10.1 -P8 -i 1 -t 1000

    Connecting to host 192.168.10.1, port 5201

    [  4] local 192.168.168.1 port 38052 connected to 192.168.10.1 port 5201

    [  6] local 192.168.168.1 port 38054 connected to 192.168.10.1 port 5201

    [  8] local 192.168.168.1 port 38056 connected to 192.168.10.1 port 5201

    [ 10] local 192.168.168.1 port 38058 connected to 192.168.10.1 port 5201

    [ 12] local 192.168.168.1 port 38060 connected to 192.168.10.1 port 5201

    [ 14] local 192.168.168.1 port 38062 connected to 192.168.10.1 port 5201

    [ 16] local 192.168.168.1 port 38064 connected to 192.168.10.1 port 5201

    [ 18] local 192.168.168.1 port 38066 connected to 192.168.10.1 port 5201

    [ ID] Interval Transfer     Bandwidth       Retr Cwnd

    [  4]   0.00-1.00 sec   764 MBytes  6.41 Gbits/sec    0 306 KBytes

    [  6]   0.00-1.00 sec   764 MBytes  6.41 Gbits/sec    0 315 KBytes

    [  8]   0.00-1.00 sec   765 MBytes  6.42 Gbits/sec    0 306 KBytes

    [ 10]   0.00-1.00   sec 765 MBytes  6.42 Gbits/sec    0 297 KBytes

    [ 12]   0.00-1.00   sec 765 MBytes  6.42 Gbits/sec    0 271 KBytes

    [ 14]   0.00-1.00   sec 763 MBytes  6.40 Gbits/sec    0 253 KBytes

    [ 16]   0.00-1.00   sec 764 MBytes  6.41 Gbits/sec    0 315 KBytes

    [ 18]   0.00-1.00   sec 764 MBytes  6.40 Gbits/sec    0 280 KBytes

    [SUM]   0.00-1.00   sec 5.97 GBytes  51.3 Gbits/sec    0