Configuring EVPN with BGP Unnumbered on Spectrum Switches

Version 6

    This post shows how to configure EVPN over BGP unnumbered using Mellanox Spectrum switches installed with Cumulus Linux.

     

    References

     

    Setup

    The following setup consists of four switches configured a unnumbered BGP routers. See HowTo Configure BGP Unnumbered on Cumulus Linux for more details.

     

    Topology with BGP:

     

    To this topology, we created VXLAN tunnels configured over this L3 network.

     

    Topology with VXLAN configs:

    Configuration

    spine - 1

     

    /etc/network/interfaces

    # This file describes the network interfaces available on your system

    # and how to activate them. For more information, see interfaces(5).

     

    source /etc/network/interfaces.d/*.intf

     

    # The loopback network interface

    auto lo

    iface lo inet loopback

        address 1.1.1.1/32

        address 10.10.10.1/32

     

    # The primary network interface

    auto eth0

    iface eth0 inet dhcp

     

    auto swp1

    iface swp1

     

    auto swp2

    iface swp2

     

    /etc/quagga/Quagga.conf

    username cumulus nopassword

    !

    service integrated-vtysh-config

    !

    log file /var/log/quagga/quagga.log

    !

    log timestamp precision 6

    !

    interface swp1

    ipv6 nd ra-interval 5

    no ipv6 nd suppress-ra

    !

    interface swp2

    ipv6 nd ra-interval 5

    no ipv6 nd suppress-ra

    !

    router bgp 65000

    bgp router-id 1.1.1.1

    bgp bestpath as-path multipath-relax

    bgp bestpath compare-routerid

    neighbor fabric peer-group

    neighbor fabric remote-as external

    neighbor fabric description Internal Fabric Network

    neighbor fabric capability extended-nexthop

    neighbor swp1 interface peer-group fabric

    neighbor swp2 interface peer-group fabric

    !

    address-family ipv4 unicast

      network 10.10.10.1/32

    exit-address-family

    !

    address-family ipv6 unicast

      neighbor fabric activate

    exit-address-family

    !

    address-family evpn

      neighbor fabric activate

      neighbor swp2 activate

      neighbor swp1 activate

    exit-address-family

    exit

    !

    line vty

     

    spine - 2

    /etc/network/interfaces

    # This file describes the network interfaces available on your system

    # and how to activate them. For more information, see interfaces(5).

     

    source /etc/network/interfaces.d/*.intf

     

    # The loopback network interface

    auto lo

    iface lo inet loopback

        address 1.1.1.2/32

        address 10.10.10.2/32

     

    # The primary network interface

    auto eth0

    iface eth0 inet dhcp

     

    auto swp1

    iface swp1

        link-speed 40000

     

    auto swp2

    iface swp2

     

    /etc/quagga/Quagga.conf

    username cumulus nopassword

    !

    service integrated-vtysh-config

    !

    log file /var/log/quagga/quagga.log

    !

    log timestamp precision 6

    !

    interface swp1

    ipv6 nd ra-interval 5

    no ipv6 nd suppress-ra

    !

    interface swp2

    ipv6 nd ra-interval 5

    no ipv6 nd suppress-ra

    !

    router bgp 65000

    bgp router-id 1.1.1.2

    bgp bestpath as-path multipath-relax

    bgp bestpath compare-routerid

    neighbor fabric peer-group

    neighbor fabric remote-as external

    neighbor fabric description Internal Fabric Network

    neighbor fabric capability extended-nexthop

    neighbor swp1 interface peer-group fabric

    neighbor swp2 interface peer-group fabric

    !

    address-family ipv4 unicast

      network 10.10.10.2/32

    exit-address-family

    !

    address-family ipv6 unicast

      neighbor fabric activate

    exit-address-family

    !

    address-family evpn

      neighbor fabric activate

      neighbor swp2 activate

      neighbor swp1 activate

    exit-address-family

    exit

    !

     

     

    line vty

     

    Leaf - 1

    /etc/network/interfaces

    # This file describes the network interfaces available on your system

    # and how to activate them. For more information, see interfaces(5).

     

    source /etc/network/interfaces.d/*.intf

     

    # The loopback network interface

    auto lo

    iface lo inet loopback

        address 2.2.2.1/32

        address 20.20.20.1/32

     

    # The primary network interface

    auto eth0

    iface eth0 inet dhcp

     

    auto swp1

    iface swp1

     

    auto swp12

    iface swp12

        link-speed 40000

     

    auto swp16

    iface swp16

        bridge-access 1000

     

    auto bridge

    iface bridge

        bridge-ports swp16 vxlan10001

        bridge-vids 1 1000

        bridge-vlan-aware yes

        post-up bridge fdb add 0:00:10:00:00:0C dev vxlan10001 dst 20.20.20.2 vni 10001

     

    auto vlan1000

    iface vlan1000

        address 100.100.100.1/24

        vlan-id 1000

        vlan-raw-device bridge

     

    auto vxlan10001

    iface vxlan10001

        vxlan-id 10001

        vxlan-local-tunnelip 20.20.20.1

        bridge-access 1000

         bridge-learning off

    /etc/quagga/Quagga.conf

    username cumulus nopassword

    !

    service integrated-vtysh-config

    !

    debug bgp updates prefix 10.10.10.2/32

    !

    log file /var/log/quagga/quagga.log

    !

    log timestamp precision 6

    !

    interface swp1

    ipv6 nd ra-interval 5

    no ipv6 nd suppress-ra

    !

    interface swp12

    ipv6 nd ra-interval 5

    no ipv6 nd suppress-ra

    !

    router bgp 65002

    bgp router-id 2.2.2.1

    bgp bestpath as-path multipath-relax

    bgp bestpath compare-routerid

    neighbor fabric peer-group

    neighbor fabric remote-as external

    neighbor fabric description Internal Fabric Network

    neighbor fabric capability extended-nexthop

    neighbor swp1 interface peer-group fabric

    neighbor swp12 interface peer-group fabric

    !

    address-family ipv4 unicast

      network 20.20.20.1/32

    exit-address-family

    !

    address-family ipv6 unicast

      neighbor fabric activate

    exit-address-family

    !

    address-family evpn

      neighbor fabric activate

      neighbor swp1 activate

      neighbor swp12 activate

      advertise-all-vni

    exit-address-family

    exit

    !

     

    Leaf - 2

    /etc/network/interfaces

    source /etc/network/interfaces.d/*.intf

     

    # The loopback network interface

    auto lo

    iface lo inet loopback

        address 2.2.2.2/32

        address 20.20.20.2/32

     

    # The primary network interface

    auto eth0

    iface eth0 inet dhcp

     

    iface eth1 inet dhcp

     

    auto swp1

    iface swp1

        bridge-access 1000

     

    auto swp21

    iface swp21

     

    auto swp22

    iface swp22

    auto swp26

    iface swp26

     

    auto bridge

    iface bridge

        bridge-ports swp1 vxlan10001

        bridge-vids 1 1000

        bridge-vlan-aware yes

        post-up bridge fdb add 0:00:10:22:00:0C dev vxlan10001 dst 20.20.20.1 vni 10001

     

    auto vlan1000

    iface vlan1000

        address 100.100.100.2/24

        vlan-id 1000

        vlan-raw-device bridge

     

    auto vxlan10001

    iface vxlan10001

        vxlan-id 10001

        vxlan-local-tunnelip 20.20.20.2

        bridge-access 1000

        bridge-learning off

     

    /etc/quagga/Quagga.conf

    username cumulus nopassword

    !

    service integrated-vtysh-config

    !

    debug bgp updates prefix 10.10.10.2/32

    !

    log file /var/log/quagga/quagga.log

    !

    log timestamp precision 6

    !

    interface swp21

    ipv6 nd ra-interval 5

    no ipv6 nd suppress-ra

    !

    interface swp22

    ipv6 nd ra-interval 5

    no ipv6 nd suppress-ra

    !

    router bgp 65003

    bgp router-id 2.2.2.2

    bgp bestpath as-path multipath-relax

    bgp bestpath compare-routerid

    neighbor fabric peer-group

    neighbor fabric remote-as external

    neighbor fabric description Internal Fabric Network

    neighbor fabric capability extended-nexthop

    neighbor swp21 interface peer-group fabric

    neighbor swp22 interface peer-group fabric

    !

    address-family ipv4 unicast

      network 20.20.20.2/32

    exit-address-family

    !

    address-family ipv6 unicast

      neighbor fabric activate

    exit-address-family

    !

    address-family evpn

      neighbor fabric activate

      neighbor swp21 activate

      neighbor swp22 activate

      advertise-all-vni

    exit-address-family

    exit

    !

     

     

    line vty

     

    Verification

     

    BGP Control Plane

     

    Spine - 1

    cumulus@cumulus:~$ net show route

     

    show ip route

    =============

    Codes: K - kernel route, C - connected, S - static, R - RIP,

           O - OSPF, I - IS-IS, B - BGP, P - PIM, T - Table, v - VNC,

           V - VPN,

           > - selected route, * - FIB route

     

    K>* 0.0.0.0/0 via 10.20.0.251, eth0

    C>* 1.1.1.1/32 is directly connected, lo

    C>* 10.10.10.1/32 is directly connected, lo

    C>* 10.20.0.0/16 is directly connected, eth0

    B>* 20.20.20.1/32 [20/0] via fe80::7efe:90ff:fefc:7bd8, swp1, 00:22:35

    B>* 20.20.20.2/32 [20/0] via fe80::7efe:90ff:fe79:c808, swp2, 00:23:27

     

    show ipv6 route

    ===============

    Codes: K - kernel route, C - connected, S - static, R - RIPng,

           O - OSPFv6, I - IS-IS, B - BGP, T - Table, v - VNC,

           V - VPN,

           > - selected route, * - FIB route

     

    C * fe80::/64 is directly connected, swp1

    C * fe80::/64 is directly connected, swp2

    C * fe80::/64 is directly connected, swid0_eth

    C>* fe80::/64 is directly connected, eth0

    Spine - 2

    cumulus@cumulus:~$ net show route

     

    show ip route

    =============

    Codes: K - kernel route, C - connected, S - static, R - RIP,

           O - OSPF, I - IS-IS, B - BGP, P - PIM, T - Table, v - VNC,

           V - VPN,

           > - selected route, * - FIB route

     

    K>* 0.0.0.0/0 via 10.20.0.251, eth0

    C>* 1.1.1.2/32 is directly connected, lo

    C>* 10.10.10.2/32 is directly connected, lo

    C>* 10.20.0.0/16 is directly connected, eth0

    B>* 20.20.20.1/32 [20/0] via fe80::7efe:90ff:fefc:7be8, swp1, 00:23:19

    B>* 20.20.20.2/32 [20/0] via fe80::7efe:90ff:fe79:c80a, swp2, 00:24:11

     

    show ipv6 route

    ===============

    Codes: K - kernel route, C - connected, S - static, R - RIPng,

           O - OSPFv6, I - IS-IS, B - BGP, T - Table, v - VNC,

           V - VPN,

           > - selected route, * - FIB route

     

    C * fe80::/64 is directly connected, eth0

    C * fe80::/64 is directly connected, swp1

    C * fe80::/64 is directly connected, swp2

    C>* fe80::/64 is directly connected, swid0_eth

     

    Leaf - 1

    cumulus@cumulus:~$ net show route

     

    show ip route

    =============

    Codes: K - kernel route, C - connected, S - static, R - RIP,

           O - OSPF, I - IS-IS, B - BGP, P - PIM, T - Table, v - VNC,

           V - VPN,

           > - selected route, * - FIB route

     

    K>* 0.0.0.0/0 via 10.20.0.251, eth0

    C>* 2.2.2.1/32 is directly connected, lo

    B>* 10.10.10.1/32 [20/0] via fe80::7efe:90ff:fefc:7ad8, swp1, 00:24:01

    B>* 10.10.10.2/32 [20/0] via fe80::7efe:90ff:fefc:7c58, swp12, 00:24:00

    C>* 10.20.0.0/16 is directly connected, eth0

    C>* 20.20.20.1/32 is directly connected, lo

    B>* 20.20.20.2/32 [20/0] via fe80::7efe:90ff:fefc:7ad8, swp1, 00:24:00

      *                      via fe80::7efe:90ff:fefc:7c58, swp12, 00:24:00

    C>* 100.100.100.0/24 is directly connected, vlan1000

     

    show ipv6 route

    ===============

    Codes: K - kernel route, C - connected, S - static, R - RIPng,

           O - OSPFv6, I - IS-IS, B - BGP, T - Table, v - VNC,

           V - VPN,

           > - selected route, * - FIB route

     

    C * fe80::/64 is directly connected, vlan1000

    C * fe80::/64 is directly connected, bridge

    C * fe80::/64 is directly connected, swid0_eth

    C * fe80::/64 is directly connected, swp12

    C * fe80::/64 is directly connected, swp1

    C>* fe80::/64 is directly connected, eth0

     

    Leaf - 2

    cumulus@cumulus:~$ net show route

     

    show ip route

    =============

    Codes: K - kernel route, C - connected, S - static, R - RIP,

           O - OSPF, I - IS-IS, B - BGP, P - PIM, T - Table, v - VNC,

           V - VPN,

           > - selected route, * - FIB route

     

    K>* 0.0.0.0/0 via 10.20.0.251, eth0

    C>* 2.2.2.2/32 is directly connected, lo

    B>* 10.10.10.1/32 [20/0] via fe80::7efe:90ff:fefc:7adc, swp21, 00:25:40

    B>* 10.10.10.2/32 [20/0] via fe80::7efe:90ff:fefc:7c5c, swp22, 00:25:39

    C>* 10.20.0.0/16 is directly connected, eth0

    B>* 20.20.20.1/32 [20/0] via fe80::7efe:90ff:fefc:7adc, swp21, 00:24:47

      *                      via fe80::7efe:90ff:fefc:7c5c, swp22, 00:24:47

    C>* 20.20.20.2/32 is directly connected, lo

    C>* 100.100.100.0/24 is directly connected, vlan1000

     

     

    show ipv6 route

    ===============

    Codes: K - kernel route, C - connected, S - static, R - RIPng,

           O - OSPFv6, I - IS-IS, B - BGP, T - Table, v - VNC,

           V - VPN,

           > - selected route, * - FIB route

     

    C * fe80::/64 is directly connected, swid0_eth

    C * fe80::/64 is directly connected, vlan1000

    C * fe80::/64 is directly connected, bridge

    C * fe80::/64 is directly connected, swp22

    C * fe80::/64 is directly connected, swp21

    C>* fe80::/64 is directly connected, eth0

     

    EVPN details

     

    Leaf - 1

    cumulus@cumulus:~$ sudo vtysh

    Hello, this is Quagga (version 1.0.0+cl3eau8).

    Copyright 1996-2005 Kunihiro Ishiguro, et al.

     

    cumulus# show bgp evpn vni

    Advertise All VNI flag: Enabled

    Number of VNIs: 1

    Flags: * - Kernel

      VNI        Orig IP         RD                    Import RT             Export RT 

    * 10001      20.20.20.1      2.2.2.1:10001         65002:10001           65002:10001

    cumulus#

     

    Leaf - 2

    cumulus@cumulus:~$ sudo vtysh

     

    Hello, this is Quagga (version 1.0.0+cl3eau8).

    Copyright 1996-2005 Kunihiro Ishiguro, et al.

     

    cumulus#

    cumulus# show bgp evpn vni

    Advertise All VNI flag: Enabled

    Number of VNIs: 1

    Flags: * - Kernel

      VNI        Orig IP         RD                    Import RT             Export RT

    * 10001      20.20.20.2      2.2.2.2:10001         65003:10001           65003:10001

    cumulus#

     

    Learned MAC

     

    Leaf - 1

    cumulus@cumulus:~$ net show bridge macs

     

    VLAN      Master    Interface    MAC                TunnelDest    State      Flags    LastSeen

    --------  --------  -----------  -----------------  ------------  ---------  -------  ----------

    1000      bridge    bridge       7c:fe:90:fc:7b:f8                permanent           02:14:33

    1000      bridge    swp16        7c:fe:90:f2:34:c1                                    00:00:02

    1000      bridge    vxlan10001   e4:1d:2d:46:f9:3c                                    00:00:02

    untagged            vxlan10001   00:00:00:00:00:00  20.20.20.2    permanent  self     00:28:57

    untagged            vxlan10001   00:00:10:00:00:0c  20.20.20.2    permanent  self     02:14:34

    untagged            vxlan10001   e4:1d:2d:46:f9:3c  20.20.20.2               self     00:28:57

    untagged  bridge    swp16        7c:fe:90:fc:7b:f8                permanent           02:14:34

    untagged  bridge    vxlan10001   9a:5a:3c:18:3e:11                permanent           02:14:34

    cumulus@cumulus:~$

     

    Leaf - 2

    cumulus@cumulus:~$ net show bridge mac

     

    VLAN      Master    Interface    MAC                TunnelDest    State      Flags    LastSeen

    --------  --------  -----------  -----------------  ------------  ---------  -------  ----------

    1000      bridge    bridge       3e:0e:05:1a:45:40                permanent           02:56:32

    1000      bridge    swp1         e4:1d:2d:46:f9:3c                                    00:00:09

    1000      bridge    vxlan10001   7c:fe:90:f2:34:c1                                    00:00:09

    untagged            vxlan10001   00:00:00:00:00:00  20.20.20.1    permanent  self     00:29:41

    untagged            vxlan10001   00:00:10:22:00:0c  20.20.20.1    permanent  self     02:56:32

    untagged            vxlan10001   7c:fe:90:f2:34:c1  20.20.20.1               self     00:29:41

    untagged  bridge    swp1         7c:fe:90:79:c8:3c                permanent           02:56:32

    untagged  bridge    vxlan10001   3e:0e:05:1a:45:40                permanent           02:56:32

    cumulus@cumulus:~$

     

    Ping Test

    from Host - 1 to Host - 2

    c-csi-sn2700-0237 [standalone: master] # show ip interface brief

    Interface         Address/Mask           Admin-state       Oper-state        MTU          VRF

    mgmt0             10.20.3.28/16          Enabled           Up                1500         default

    mgmt1             0.0.0.0/0              Enabled           Up                1500         default

    Vlan 50           0.0.0.0/0              Disabled          Down              1500         default

    Vlan 310          10.234.70.105/29       Enabled           Up                1500         default

    Eth1/1            13.0.0.13/24           Enabled           Down              1500         default

    Eth1/16           100.100.100.100/24     Enabled           Up                1500         default

    Eth1/19           0.0.0.0/0              Disabled          Down              1500         default

    c-csi-sn2700-0237 [standalone: master] # ping 100.100.100.200

    PING 100.100.100.200 (100.100.100.200) 56(84) bytes of data.

    64 bytes from 100.100.100.200: icmp_seq=1 ttl=64 time=0.501 ms

    64 bytes from 100.100.100.200: icmp_seq=2 ttl=64 time=0.520 ms

    64 bytes from 100.100.100.200: icmp_seq=3 ttl=64 time=0.539 ms

    ^C

    --- 100.100.100.200 ping statistics ---

    3 packets transmitted, 3 received, 0% packet loss, time 2819ms

    rtt min/avg/max/mdev = 0.501/0.520/0.539/0.015 ms

    c-csi-sn2700-0237 [standalone: master] #

     

    from Host - 2 to Host - 1

    cumulus@c-csi-sn2700-0336:~$ ping 100.100.100.100

    PING 100.100.100.100 (100.100.100.100) 56(84) bytes of data.

    64 bytes from 100.100.100.100: icmp_seq=1 ttl=64 time=0.521 ms

    64 bytes from 100.100.100.100: icmp_seq=2 ttl=64 time=0.515 ms

    64 bytes from 100.100.100.100: icmp_seq=3 ttl=64 time=0.532 ms

    ^C

    --- 100.100.100.100 ping statistics ---

    3 packets transmitted, 3 received, 0% packet loss, time 1999ms

    rtt min/avg/max/mdev = 0.515/0.522/0.532/0.027 ms

    cumulus@c-csi-sn2700-0336:~$