HowTo Configure VXLAN Routing using Loopback on Cumulus Linux

Version 6

    This is a configuration guide for VXLAN routing using loopback on Cumulus Linux OS over Spectrum switches.

     

    References

     

    Setup

    VXLAN routing enables communication between hosts, which are not part of the same VXLAN network or VNIs.

    The figure below illustrates the communication process between Host-1 and Host-2 under the following properties:

    • Host-1 is on 10.10.10.x/24 network and is part of VNI 2000
    • Host-2 is on 20.20.20.x/24 network and is not part of VNI 2000
    • VXLAN routing runs based on the native tenant IP packet and not the source/destination IPs of the VXLAN tunnel endpoints (VTEPs)
    • The routing process occurs before encapsulation or after decapsulation of VXLAN headers

     

     

     

     

     

     

     

     

     

     

     

     

     

     

    Initiating VXLAN routing requires a loopback wire mechanism on one of the Leafs (Leaf-2 in the illustrations).

    The figure below illustrates the loopback wire mechanism under the following configuration:

    • Host-1- part of VLAN 10 and subnet 10.10.10.11/24 (default gateway)
    • Host-2- part of VLAN 20 and subnet 20.20.20.33/24
    • Leaf-2- part of VLAN 10 is mapped to VNI 2000. VNI 2000 has 10.10.10.11 MAC address learned from VTEP 1.1.1.1

     

    Host-2's default gateway (20.20.20.x/24) enables Leaf-2 to route the packet received on VLAN 20 to swp10 (on 10.10.10.x network), and from swp10 the packet is transferred back to the switch on swp9 (part of VLAN 10) through a loopback cable.
    The packet will be encapsulated and sent to Leaf-1 then delivered to Host-1 as normal processing.

    Host-1's default gateway (10.10.10.x/24) enables the packet to be transferred towards 10.10.10.3, same IP which routes the packet to Leaf-2 and connect the two Hosts.

     

     

     

     

     

     

     

     

     

     

    Below is the mapping configuration of VLAN 10 to VXLAN 2000 which enables a two way routing of the packets on Leaf-2, from 10.10.10.x/24 to 20.20.20.x/24.
    Leaf-1 and Leaf-2 has VTEPs (the loopback address is used as VTEPs) and the Spine is the service node for VxLAN 2000.

    Configuration

    Spine

    #configure Interfaces

    net add loopback lo ip address 2.2.2.2/32

    net add loopback swp2 ip address 12.12.12.2/24

    net add loopback swp3 ip address 23.23.23.2/24

     

    #configure routing

    net add ospf router-id 2.2.2.2

    net add router ospf

    net add interface lo ip ospf area 0

    net add interface swp2 ip ospf area 0

    net add interface swp3 ip ospf area 0

    net add interface swp21 ip ospf network broadcast

    net add interface swp3 ip ospf network broadcast

    net add interface lo ip ospf network broadcast

     

    #configure Service Node functionality, along with Anycast IP

    net add lnv service-node source 2.2.2.2

    net add lnv service-node anycast-ip 2.2.2.2

    The example in this document only shows one spine, in case of multiple spines, Anycast IP is required for service replication load balancing.
    Additional configuration is required when utilizing three Spines (Spine – 2 has service IP 5.5.5.5 and Spine – 3 has service IP 6.6.6.6)

    net add lnv service-node peers 2.2.2.2 5.5.5.5 6.6.6.6

     

    Leaf-1

    #configure Interfaces

    net add loopback lo ip address 1.1.1.1/32

    net add loopback swp2 ip address 12.12.12.1/24

     

    #configure routing

    net add ospf router-id 1.1.1.1

    net add router ospf

    net add interface lo ip ospf area 0

    net add interface swp2 ip ospf area 0

    net add interface swp2 ip ospf network broadcast

    net add interface lo ip ospf network broadcast

     

    #configure VTEP and Service Node IP

    net add loopback lo vxrd-src-ip 1.1.1.1

    net add loopback lo vxrd-svcnode-ip 2.2.2.2

     

    #host VLAN configs

    net add vlan 10

    net add interface swp16 bridge-access 10

    net add vlan-interface vlan10 address 10.10.10.1/24

     

    #VLAN to VxLAN Mapping

    net add vni vni2000 bridge-access 10

    net add vni vni2000 vxlan-local-tunnelip 1.1.1.1

     

    Leaf-2

    #configure Interfaces

    net add loopback lo ip address 3.3.3.3/32

    net add loopback swp3 ip address 23.23.23.3/24

     

    #configure routing

    net add interface lo ip ospf area 0

    net add interface swp3 ip ospf area 0

    net add interface swp3 ip ospf network broadcast

    net add interface lo ip ospf network broadcast

    net add ospf router-id 3.3.3.3

    net add router ospf

     

    #configure VTEP and Service Node IP

    net add loopback lo vxrd-src-ip 3.3.3.3

    net add loopback lo vxrd-svcnode-ip 2.2.2.2

     

    #host VLAN configs (adding loopback port 1st end in VLAN 10)

    net add vlan 10

    net add interface swp9 bridge-access 10

     

    #VLAN to VxLAN Mapping

    net add vni vni2000 bridge-access 10

    net add vni vni2000 vxlan-local-tunnelip 3.3.3.3

     

    #Loopback port config (adding loopback port 2nd end in 10.10.10.x IP)

    net add interface swp10 ip address 10.10.10.3/24

     

    #routed host VLAN configs (20.20.20.x/24 VLAN 20)

    net add interface vlan20 ip address 20.20.20.3/24

    net add interface swp16 bridge-access 20

     

    Host

    #Host – 1 default GW configs
    ip route 20.20.20.0/24 10.10.10.3
    #Host – 2 default GW configs

    ip route 10.10.10.0/24 20.20.20.3

     

    Verification

    Verification can be done with show commands.
    The roles are different for Leaf and Spine switches, Leaf it is VTEP and Spine it is Service node, furthermore, Leafs have learned remote VTEPs along with local membership of mapped VLANs.
    Once the information is verified, it is possible to run a Ping test from Host-1 on Leaf-1 to Host-2 on Leaf-2

    Spine

    Leaf-1

    Leaf-2

    End to End Traceroute and Ping

    Host-1 (10.10.10.11) to Host-2 (20.20.20.33)

    Host-2 (20.20.20.33) to Host-1 (10.10.10.11)

    Learned MACs

    On Leaf-1: Host-2 MAC learned on VNI 2000 with remote VTEP as 3.3.3.3

    On Leaf-2: Host-1 MAC learned on VNI 2000 with remote VTEP as 1.1.1.1