Configuring EVPN with BGP Unnumbered on Spectrum Switches

Version 7

    This is an EVPN over BGP unnumbered configuration guide for Mellanox Spectrum switches running on Cumulus Linux.

     

    References

     

    Setup

    The following setup consists of four switches configured with unnumbered BGP routers. See HowTo Configure BGP Unnumbered on Cumulus Linux for more details.

     

    Topology with BGP unnumbered:

     

    For this topology, we created VXLAN tunnels configured over L3 network.

     

    Topology with VXLAN configurations:

    Configuration

    spine - 1

     

    /etc/network/interfaces

    # This file describes the network interfaces available on your system

    # and how to activate them. For more information, see interfaces(5).

     

    source /etc/network/interfaces.d/*.intf

     

    # The loopback network interface

    auto lo

    iface lo inet loopback

        address 1.1.1.1/32

        address 10.10.10.1/32

     

    # The primary network interface

    auto eth0

    iface eth0 inet dhcp

     

    auto swp1

    iface swp1

     

    auto swp2

    iface swp2

     

    /etc/quagga/Quagga.conf

    username cumulus nopassword

    !

    service integrated-vtysh-config

    !

    log file /var/log/quagga/quagga.log

    !

    log timestamp precision 6

    !

    interface swp1

    ipv6 nd ra-interval 5

    no ipv6 nd suppress-ra

    !

    interface swp2

    ipv6 nd ra-interval 5

    no ipv6 nd suppress-ra

    !

    router bgp 65000

    bgp router-id 1.1.1.1

    bgp bestpath as-path multipath-relax

    bgp bestpath compare-routerid

    neighbor fabric peer-group

    neighbor fabric remote-as external

    neighbor fabric description Internal Fabric Network

    neighbor fabric capability extended-nexthop

    neighbor swp1 interface peer-group fabric

    neighbor swp2 interface peer-group fabric

    !

    address-family ipv4 unicast

      network 10.10.10.1/32

    exit-address-family

    !

    address-family ipv6 unicast

      neighbor fabric activate

    exit-address-family

    !

    address-family evpn

      neighbor fabric activate

      neighbor swp2 activate

      neighbor swp1 activate

    exit-address-family

    exit

    !

    line vty

     

    spine - 2

    /etc/network/interfaces

    # This file describes the network interfaces available on your system

    # and how to activate them. For more information, see interfaces(5).

     

    source /etc/network/interfaces.d/*.intf

     

    # The loopback network interface

    auto lo

    iface lo inet loopback

        address 1.1.1.2/32

        address 10.10.10.2/32

     

    # The primary network interface

    auto eth0

    iface eth0 inet dhcp

     

    auto swp1

    iface swp1

        link-speed 40000

     

    auto swp2

    iface swp2

     

    /etc/quagga/Quagga.conf

    username cumulus nopassword

    !

    service integrated-vtysh-config

    !

    log file /var/log/quagga/quagga.log

    !

    log timestamp precision 6

    !

    interface swp1

    ipv6 nd ra-interval 5

    no ipv6 nd suppress-ra

    !

    interface swp2

    ipv6 nd ra-interval 5

    no ipv6 nd suppress-ra

    !

    router bgp 65000

    bgp router-id 1.1.1.2

    bgp bestpath as-path multipath-relax

    bgp bestpath compare-routerid

    neighbor fabric peer-group

    neighbor fabric remote-as external

    neighbor fabric description Internal Fabric Network

    neighbor fabric capability extended-nexthop

    neighbor swp1 interface peer-group fabric

    neighbor swp2 interface peer-group fabric

    !

    address-family ipv4 unicast

      network 10.10.10.2/32

    exit-address-family

    !

    address-family ipv6 unicast

      neighbor fabric activate

    exit-address-family

    !

    address-family evpn

      neighbor fabric activate

      neighbor swp2 activate

      neighbor swp1 activate

    exit-address-family

    exit

    !

     

     

    line vty

     

    Leaf - 1

    /etc/network/interfaces

    # This file describes the network interfaces available on your system

    # and how to activate them. For more information, see interfaces(5).

     

    source /etc/network/interfaces.d/*.intf

     

    # The loopback network interface

    auto lo

    iface lo inet loopback

        address 2.2.2.1/32

        address 20.20.20.1/32

     

    # The primary network interface

    auto eth0

    iface eth0 inet dhcp

     

    auto swp1

    iface swp1

     

    auto swp12

    iface swp12

        link-speed 40000

     

    auto swp16

    iface swp16

        bridge-access 1000

     

    auto bridge

    iface bridge

        bridge-ports swp16 vxlan10001

        bridge-vids 1 1000

        bridge-vlan-aware yes

        post-up bridge fdb add 0:00:10:00:00:0C dev vxlan10001 dst 20.20.20.2 vni 10001

     

    auto vlan1000

    iface vlan1000

        address 100.100.100.1/24

        vlan-id 1000

        vlan-raw-device bridge

     

    auto vxlan10001

    iface vxlan10001

        vxlan-id 10001

        vxlan-local-tunnelip 20.20.20.1

        bridge-access 1000

         bridge-learning off

    /etc/quagga/Quagga.conf

    username cumulus nopassword

    !

    service integrated-vtysh-config

    !

    debug bgp updates prefix 10.10.10.2/32

    !

    log file /var/log/quagga/quagga.log

    !

    log timestamp precision 6

    !

    interface swp1

    ipv6 nd ra-interval 5

    no ipv6 nd suppress-ra

    !

    interface swp12

    ipv6 nd ra-interval 5

    no ipv6 nd suppress-ra

    !

    router bgp 65002

    bgp router-id 2.2.2.1

    bgp bestpath as-path multipath-relax

    bgp bestpath compare-routerid

    neighbor fabric peer-group

    neighbor fabric remote-as external

    neighbor fabric description Internal Fabric Network

    neighbor fabric capability extended-nexthop

    neighbor swp1 interface peer-group fabric

    neighbor swp12 interface peer-group fabric

    !

    address-family ipv4 unicast

      network 20.20.20.1/32

    exit-address-family

    !

    address-family ipv6 unicast

      neighbor fabric activate

    exit-address-family

    !

    address-family evpn

      neighbor fabric activate

      neighbor swp1 activate

      neighbor swp12 activate

      advertise-all-vni

    exit-address-family

    exit

    !

     

    Leaf - 2

    /etc/network/interfaces

    source /etc/network/interfaces.d/*.intf

     

    # The loopback network interface

    auto lo

    iface lo inet loopback

        address 2.2.2.2/32

        address 20.20.20.2/32

     

    # The primary network interface

    auto eth0

    iface eth0 inet dhcp

     

    iface eth1 inet dhcp

     

    auto swp1

    iface swp1

        bridge-access 1000

     

    auto swp21

    iface swp21

     

    auto swp22

    iface swp22

    auto swp26

    iface swp26

     

    auto bridge

    iface bridge

        bridge-ports swp1 vxlan10001

        bridge-vids 1 1000

        bridge-vlan-aware yes

        post-up bridge fdb add 0:00:10:22:00:0C dev vxlan10001 dst 20.20.20.1 vni 10001

     

    auto vlan1000

    iface vlan1000

        address 100.100.100.2/24

        vlan-id 1000

        vlan-raw-device bridge

     

    auto vxlan10001

    iface vxlan10001

        vxlan-id 10001

        vxlan-local-tunnelip 20.20.20.2

        bridge-access 1000

        bridge-learning off

     

    /etc/quagga/Quagga.conf

    username cumulus nopassword

    !

    service integrated-vtysh-config

    !

    debug bgp updates prefix 10.10.10.2/32

    !

    log file /var/log/quagga/quagga.log

    !

    log timestamp precision 6

    !

    interface swp21

    ipv6 nd ra-interval 5

    no ipv6 nd suppress-ra

    !

    interface swp22

    ipv6 nd ra-interval 5

    no ipv6 nd suppress-ra

    !

    router bgp 65003

    bgp router-id 2.2.2.2

    bgp bestpath as-path multipath-relax

    bgp bestpath compare-routerid

    neighbor fabric peer-group

    neighbor fabric remote-as external

    neighbor fabric description Internal Fabric Network

    neighbor fabric capability extended-nexthop

    neighbor swp21 interface peer-group fabric

    neighbor swp22 interface peer-group fabric

    !

    address-family ipv4 unicast

      network 20.20.20.2/32

    exit-address-family

    !

    address-family ipv6 unicast

      neighbor fabric activate

    exit-address-family

    !

    address-family evpn

      neighbor fabric activate

      neighbor swp21 activate

      neighbor swp22 activate

      advertise-all-vni

    exit-address-family

    exit

    !

     

     

    line vty

     

    Verification

     

    BGP Control Plane

     

    Spine - 1

    cumulus@cumulus:~$ net show route

     

    show ip route

    =============

    Codes: K - kernel route, C - connected, S - static, R - RIP,

           O - OSPF, I - IS-IS, B - BGP, P - PIM, T - Table, v - VNC,

           V - VPN,

           > - selected route, * - FIB route

     

    K>* 0.0.0.0/0 via 10.20.0.251, eth0

    C>* 1.1.1.1/32 is directly connected, lo

    C>* 10.10.10.1/32 is directly connected, lo

    C>* 10.20.0.0/16 is directly connected, eth0

    B>* 20.20.20.1/32 [20/0] via fe80::7efe:90ff:fefc:7bd8, swp1, 00:22:35

    B>* 20.20.20.2/32 [20/0] via fe80::7efe:90ff:fe79:c808, swp2, 00:23:27

     

    show ipv6 route

    ===============

    Codes: K - kernel route, C - connected, S - static, R - RIPng,

           O - OSPFv6, I - IS-IS, B - BGP, T - Table, v - VNC,

           V - VPN,

           > - selected route, * - FIB route

     

    C * fe80::/64 is directly connected, swp1

    C * fe80::/64 is directly connected, swp2

    C * fe80::/64 is directly connected, swid0_eth

    C>* fe80::/64 is directly connected, eth0

    Spine - 2

    cumulus@cumulus:~$ net show route

     

    show ip route

    =============

    Codes: K - kernel route, C - connected, S - static, R - RIP,

           O - OSPF, I - IS-IS, B - BGP, P - PIM, T - Table, v - VNC,

           V - VPN,

           > - selected route, * - FIB route

     

    K>* 0.0.0.0/0 via 10.20.0.251, eth0

    C>* 1.1.1.2/32 is directly connected, lo

    C>* 10.10.10.2/32 is directly connected, lo

    C>* 10.20.0.0/16 is directly connected, eth0

    B>* 20.20.20.1/32 [20/0] via fe80::7efe:90ff:fefc:7be8, swp1, 00:23:19

    B>* 20.20.20.2/32 [20/0] via fe80::7efe:90ff:fe79:c80a, swp2, 00:24:11

     

    show ipv6 route

    ===============

    Codes: K - kernel route, C - connected, S - static, R - RIPng,

           O - OSPFv6, I - IS-IS, B - BGP, T - Table, v - VNC,

           V - VPN,

           > - selected route, * - FIB route

     

    C * fe80::/64 is directly connected, eth0

    C * fe80::/64 is directly connected, swp1

    C * fe80::/64 is directly connected, swp2

    C>* fe80::/64 is directly connected, swid0_eth

     

    Leaf - 1

    cumulus@cumulus:~$ net show route

     

    show ip route

    =============

    Codes: K - kernel route, C - connected, S - static, R - RIP,

           O - OSPF, I - IS-IS, B - BGP, P - PIM, T - Table, v - VNC,

           V - VPN,

           > - selected route, * - FIB route

     

    K>* 0.0.0.0/0 via 10.20.0.251, eth0

    C>* 2.2.2.1/32 is directly connected, lo

    B>* 10.10.10.1/32 [20/0] via fe80::7efe:90ff:fefc:7ad8, swp1, 00:24:01

    B>* 10.10.10.2/32 [20/0] via fe80::7efe:90ff:fefc:7c58, swp12, 00:24:00

    C>* 10.20.0.0/16 is directly connected, eth0

    C>* 20.20.20.1/32 is directly connected, lo

    B>* 20.20.20.2/32 [20/0] via fe80::7efe:90ff:fefc:7ad8, swp1, 00:24:00

      *                      via fe80::7efe:90ff:fefc:7c58, swp12, 00:24:00

    C>* 100.100.100.0/24 is directly connected, vlan1000

     

    show ipv6 route

    ===============

    Codes: K - kernel route, C - connected, S - static, R - RIPng,

           O - OSPFv6, I - IS-IS, B - BGP, T - Table, v - VNC,

           V - VPN,

           > - selected route, * - FIB route

     

    C * fe80::/64 is directly connected, vlan1000

    C * fe80::/64 is directly connected, bridge

    C * fe80::/64 is directly connected, swid0_eth

    C * fe80::/64 is directly connected, swp12

    C * fe80::/64 is directly connected, swp1

    C>* fe80::/64 is directly connected, eth0

     

    Leaf - 2

    cumulus@cumulus:~$ net show route

     

    show ip route

    =============

    Codes: K - kernel route, C - connected, S - static, R - RIP,

           O - OSPF, I - IS-IS, B - BGP, P - PIM, T - Table, v - VNC,

           V - VPN,

           > - selected route, * - FIB route

     

    K>* 0.0.0.0/0 via 10.20.0.251, eth0

    C>* 2.2.2.2/32 is directly connected, lo

    B>* 10.10.10.1/32 [20/0] via fe80::7efe:90ff:fefc:7adc, swp21, 00:25:40

    B>* 10.10.10.2/32 [20/0] via fe80::7efe:90ff:fefc:7c5c, swp22, 00:25:39

    C>* 10.20.0.0/16 is directly connected, eth0

    B>* 20.20.20.1/32 [20/0] via fe80::7efe:90ff:fefc:7adc, swp21, 00:24:47

      *                      via fe80::7efe:90ff:fefc:7c5c, swp22, 00:24:47

    C>* 20.20.20.2/32 is directly connected, lo

    C>* 100.100.100.0/24 is directly connected, vlan1000

     

     

    show ipv6 route

    ===============

    Codes: K - kernel route, C - connected, S - static, R - RIPng,

           O - OSPFv6, I - IS-IS, B - BGP, T - Table, v - VNC,

           V - VPN,

           > - selected route, * - FIB route

     

    C * fe80::/64 is directly connected, swid0_eth

    C * fe80::/64 is directly connected, vlan1000

    C * fe80::/64 is directly connected, bridge

    C * fe80::/64 is directly connected, swp22

    C * fe80::/64 is directly connected, swp21

    C>* fe80::/64 is directly connected, eth0

     

    EVPN details

     

    Leaf - 1

    cumulus@cumulus:~$ sudo vtysh

    Hello, this is Quagga (version 1.0.0+cl3eau8).

    Copyright 1996-2005 Kunihiro Ishiguro, et al.

     

    cumulus# show bgp evpn vni

    Advertise All VNI flag: Enabled

    Number of VNIs: 1

    Flags: * - Kernel

      VNI        Orig IP         RD                    Import RT             Export RT 

    * 10001      20.20.20.1      2.2.2.1:10001         65002:10001           65002:10001

    cumulus#

     

    Leaf - 2

    cumulus@cumulus:~$ sudo vtysh

     

    Hello, this is Quagga (version 1.0.0+cl3eau8).

    Copyright 1996-2005 Kunihiro Ishiguro, et al.

     

    cumulus#

    cumulus# show bgp evpn vni

    Advertise All VNI flag: Enabled

    Number of VNIs: 1

    Flags: * - Kernel

      VNI        Orig IP         RD                    Import RT             Export RT

    * 10001      20.20.20.2      2.2.2.2:10001         65003:10001           65003:10001

    cumulus#

     

    Learned MAC

     

    Leaf - 1

    cumulus@cumulus:~$ net show bridge macs

     

    VLAN      Master    Interface    MAC                TunnelDest    State      Flags    LastSeen

    --------  --------  -----------  -----------------  ------------  ---------  -------  ----------

    1000      bridge    bridge       7c:fe:90:fc:7b:f8                permanent           02:14:33

    1000      bridge    swp16        7c:fe:90:f2:34:c1                                    00:00:02

    1000      bridge    vxlan10001   e4:1d:2d:46:f9:3c                                    00:00:02

    untagged            vxlan10001   00:00:00:00:00:00  20.20.20.2    permanent  self     00:28:57

    untagged            vxlan10001   00:00:10:00:00:0c  20.20.20.2    permanent  self     02:14:34

    untagged            vxlan10001   e4:1d:2d:46:f9:3c  20.20.20.2               self     00:28:57

    untagged  bridge    swp16        7c:fe:90:fc:7b:f8                permanent           02:14:34

    untagged  bridge    vxlan10001   9a:5a:3c:18:3e:11                permanent           02:14:34

    cumulus@cumulus:~$

     

    Leaf - 2

    cumulus@cumulus:~$ net show bridge mac

     

    VLAN      Master    Interface    MAC                TunnelDest    State      Flags    LastSeen

    --------  --------  -----------  -----------------  ------------  ---------  -------  ----------

    1000      bridge    bridge       3e:0e:05:1a:45:40                permanent           02:56:32

    1000      bridge    swp1         e4:1d:2d:46:f9:3c                                    00:00:09

    1000      bridge    vxlan10001   7c:fe:90:f2:34:c1                                    00:00:09

    untagged            vxlan10001   00:00:00:00:00:00  20.20.20.1    permanent  self     00:29:41

    untagged            vxlan10001   00:00:10:22:00:0c  20.20.20.1    permanent  self     02:56:32

    untagged            vxlan10001   7c:fe:90:f2:34:c1  20.20.20.1               self     00:29:41

    untagged  bridge    swp1         7c:fe:90:79:c8:3c                permanent           02:56:32

    untagged  bridge    vxlan10001   3e:0e:05:1a:45:40                permanent           02:56:32

    cumulus@cumulus:~$

     

    Ping Test

    from Host - 1 to Host - 2

    c-csi-sn2700-0237 [standalone: master] # show ip interface brief

    Interface         Address/Mask           Admin-state       Oper-state        MTU          VRF

    mgmt0             10.20.3.28/16          Enabled           Up                1500         default

    mgmt1             0.0.0.0/0              Enabled           Up                1500         default

    Vlan 50           0.0.0.0/0              Disabled          Down              1500         default

    Vlan 310          10.234.70.105/29       Enabled           Up                1500         default

    Eth1/1            13.0.0.13/24           Enabled           Down              1500         default

    Eth1/16           100.100.100.100/24     Enabled           Up                1500         default

    Eth1/19           0.0.0.0/0              Disabled          Down              1500         default

    c-csi-sn2700-0237 [standalone: master] # ping 100.100.100.200

    PING 100.100.100.200 (100.100.100.200) 56(84) bytes of data.

    64 bytes from 100.100.100.200: icmp_seq=1 ttl=64 time=0.501 ms

    64 bytes from 100.100.100.200: icmp_seq=2 ttl=64 time=0.520 ms

    64 bytes from 100.100.100.200: icmp_seq=3 ttl=64 time=0.539 ms

    ^C

    --- 100.100.100.200 ping statistics ---

    3 packets transmitted, 3 received, 0% packet loss, time 2819ms

    rtt min/avg/max/mdev = 0.501/0.520/0.539/0.015 ms

    c-csi-sn2700-0237 [standalone: master] #

     

    from Host - 2 to Host - 1

    cumulus@c-csi-sn2700-0336:~$ ping 100.100.100.100

    PING 100.100.100.100 (100.100.100.100) 56(84) bytes of data.

    64 bytes from 100.100.100.100: icmp_seq=1 ttl=64 time=0.521 ms

    64 bytes from 100.100.100.100: icmp_seq=2 ttl=64 time=0.515 ms

    64 bytes from 100.100.100.100: icmp_seq=3 ttl=64 time=0.532 ms

    ^C

    --- 100.100.100.100 ping statistics ---

    3 packets transmitted, 3 received, 0% packet loss, time 1999ms

    rtt min/avg/max/mdev = 0.515/0.522/0.532/0.027 ms

    cumulus@c-csi-sn2700-0336:~$