HowTo Configure Openstack L2 Gateway with Mellanox Spectrum Switch (VTEP)

Version 10

    This post shows how to configure a OpenStack L2 Gateway setup using Mellanox Spectrum Switch.





    L2 Gateway is a service plugin to be added to the OpenStack Networking (Neutron) services. L2 Gateway is an entity or resource which bridges two L2 domains (or networks) to achieve one seamless L2 broadcast domain,

    In the following use case L2 gateway bridges a VXLAN network and a VLAN network as shown in the figure below.


    In the figure we can see the assigned IPs for each component. The L2 Gateway agent will configure the hardware VTEP database located at the switch according to the OpenStack network topology and allocated VMs.

    All the configurations and commands used to setup the network are detailed below.


    Note: The assigned IP for the  Bare metal machine is in the same subnet as the IP assigned to the Openstack VMs (10.0.0.x in the example). In addition, the IP assigned to OpenStack compute node port is in the same subnet as the switch port that it is connected to  (2.2.2.x in the example).


    The control plan (configuration) is done via OVS DB agent on the switch being address with Jason API from OpenStack Neutron CLI. The Neutron creates Bare Metal (BM) server ports on the switch and attach them to the VM network (over VXLAN), this reaches the switch in Jason API (not CLI configuration). each VXLAN tunnel is mapped to VLAN+port on a switch.


    MAC Address table

    On the one direction from the BM server to the VMs the switch holds the MACs of the VMs and answer upon ARP request. You can think about it as a static configuration of ARP entries for each VM.

    On the other direction from the VMs to the BM server there is a broadcast of the ARP for the VMs to learn (dynamic).








    For this setup a Mellanox Spectrum switch was used with image version 3.6.3502 or later, and Openstack Ocata release.


    L2 Gateway installation

    L2 Gateway agent can be installed using PIP or as used here with DevStack. The project is located here: GitHub - openstack/networking-l2gw: API's and implementations to support L2 Gateways in Neutron.

    In order to clone it, add the following line to local.conf file.

    enable_plugin networking-l2gw


    Additional information about L2 Gateway background and installation can be found here:


    Switch Configuration

    switch (config) # enable
    switch (config) # configure terminal
    switch (config) # interface loopback 1 ip address              --> create a loopback device to receive the Vxlan l3 tunneled packet
    switch (config) # ip routing vrf default                                  --> enable ip routing
    switch (config) # protocol nve                                            --> enable nve protocol
    switch (config) # interface nve 1                                         --> create A nve interface
    switch (config interface nve 1) vxlan source interface loopback 1         --> set the interface to handle vxlan with loopback interface 1
    switch (config) # interface ethernet 1/3 nve mode only force              --> Set the BM port to be associated with nve
    switch (config) # interface ethernet 1/4 no switchport force              --> Set the vxlan port to be A router port
    switch (config) # interface ethernet 1/4 ip address --> assign ip to the vxlan port (same subnet as the vxlan tunnel endpoint)
    switch (config) # ovs ovsdb server                                        --> start ovsdb

    switch (config) # ovs ovsdb server listen tcp port 6640                   --> set ovs server listen port
    switch (config) # write memory                                            --> write configuration to memory


    Bare-metal interface configuration

    Create vlan interface and assign an IP interface to the the same subnet as the OpenStack VM:

    # ip link add link eth0 name ens3f0.8 type vlan id 8

    # ip addr add dev ens3f0.8


    Openstack configuration

    Configure OpenStack compute interface connected to the switch as follows:
    1. Assign an IP in the same subnet of the IP assigned to the connected switch port we assigned in the previous section and route the loopback IP through the interface connected to the switch.

    # ip addr add dev enp3s0f0
    # route add -net netmask gw


    2. The following commands update the switch's hardware vtep DB:

    In those commands we create L2 Gateway called vtep0 for the Bare metal server port (eth3) in the switch MLNX-GW-ETH3. In addition, we map the VLAN 8 to the private network of the VMs (VNI).

    # neutron l2-gateway-create --device name="vtep0",interface_names="eth3" MLNX-GW-ETH3
    # neutron l2-gateway-connection-create --default-segmentation-id 8 MLNX-GW-ETH3 private


    3. Add a security group rule allowing ICMP packets to reach the VM:

    # nova secgroup-add-rule default icmp -1 -1




    # Switch to Neutron
    disable_service n-net
    enable_service n-cond
    enable_service q-svc
    enable_service q-dhcp
    enable_service q-l3
    enable_service q-meta
    enable_service neutron 

    # enable compute node services
    enable_service n-cpu
    enable_service q-agt 

    # Disable cinder/heat/tempest for faster testing
    disable_service c-sch c-api c-vol h-eng h-api h-api-cfn h-api-cw tempest 

    # Secrets

    ## Use ML2 + OVS

    # Set ML2 plugin + OVS agent

    # Set ML2 mechanism drivers to OVS

    # Set type drivers

    # Use Neutron security groups

    # Set possible tenant network types

    # Simple GRE tunnel configuration -- overrides extra opts

    # L2GW
    enable_plugin networking-l2gw
    enable_service l2gw-plugin l2gw-agent


    l2_population = True
    enable_vxlan = True 
    local_ip = 
    scheduler_default_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, PciPassthroughFilter
    pci_passthrough_whitelist ={"'"address"'":"'"*:'"${mlnx_dev}"'.*"'","'"physical_network"'":"'"default"'"}


    How to validate the L2 Gateway installation

    To dump the hardware vteo database run the following command from any machine with network access the the switch:

    # ovsdb-client dump --pretty tcp:<switch_ip>:6640 hardware_vtep


    The output should look similar to the following (depending on the assigned IPs and number of ports):

    id acl_entries acl_fault_status acl_name

    ----- ----------- ---------------- --------



    ACL_entry table

    _uuid acle_fault_status action dest_ip dest_mac dest_mask dest_port_max dest_port_min direction ethertype icmp_code icmp_type protocol sequence source_ip source_mac source_mask source_port_max source_port_min tcp_flags tcp_flags_mask

    ----- ----------------- ------ ------- -------- --------- ------------- ------------- --------- --------- --------- --------- -------- -------- --------- ---------- ----------- --------------- --------------- --------- --------------



    Arp_Sources_Local table

    _uuid locator src_mac

    ----- ------- -------



    Arp_Sources_Remote table

    _uuid locator src_mac

    ----- ------- -------



    Global table

    _uuid                                managers switches                             

    ------------------------------------ -------- --------------------------------------

    47927954-5124-4a18-9434-049f7f41a5b7 []       [da9f2821-09f9-4c2f-beb9-4174de6fdd3b]



    Logical_Binding_Stats table

    _uuid                                bytes_from_local bytes_to_local packets_from_local packets_to_local

    ------------------------------------ ---------------- -------------- ------------------ ----------------

    ef1dc78d-ac5c-4804-b3e9-649f2d435e9c 2822346          5709962        20764              88835          



    Logical_Router table

    LR_fault_status _uuid acl_binding description name static_routes switch_binding

    --------------- ----- ----------- ----------- ---- ------------- --------------



    Logical_Switch table

    _uuid                                description name                                   tunnel_key

    ------------------------------------ ----------- -------------------------------------- ----------

    128cf0c6-dfc7-47ce-8d4d-cc29b672ba34 private     "6be6cf3b-56c1-4f9e-b9e1-3a2f98e7bf06" 33       



    Manager table

    _uuid inactivity_probe is_connected max_backoff other_config status target

    ----- ---------------- ------------ ----------- ------------ ------ ------



    Mcast_Macs_Local table

    MAC         _uuid                                ipaddr locator_set                          logical_switch                     

    ----------- ------------------------------------ ------ ------------------------------------ ------------------------------------

    unknown-dst a39cdd30-3817-4a6f-abb9-7627bc0360e8 ""     bc94782a-17cd-441e-a7f9-3bca39b49496 128cf0c6-dfc7-47ce-8d4d-cc29b672ba34



    Mcast_Macs_Remote table

    MAC         _uuid                                ipaddr locator_set                          logical_switch                     

    ----------- ------------------------------------ ------ ------------------------------------ ------------------------------------

    unknown-dst b098a853-0da3-44ec-8cea-c57b86bce530 ""     1e060e3f-c7b7-4885-94bb-ba5b063ce20d 128cf0c6-dfc7-47ce-8d4d-cc29b672ba34



    Physical_Locator table

    _uuid                                dst_ip    encapsulation_type

    ------------------------------------ --------- ------------------

    e5b3826b-0005-4cfa-8bac-e9f693cc3715 "" "vxlan_over_ipv4"

    82bcef2f-1a8e-4191-b4b2-3255fdd6849c "" "vxlan_over_ipv4"



    Physical_Locator_Set table

    _uuid                                locators                             

    ------------------------------------ --------------------------------------

    1e060e3f-c7b7-4885-94bb-ba5b063ce20d [82bcef2f-1a8e-4191-b4b2-3255fdd6849c]

    bc94782a-17cd-441e-a7f9-3bca39b49496 [e5b3826b-0005-4cfa-8bac-e9f693cc3715]



    Physical_Port table

    _uuid                                acl_bindings description name   port_fault_status vlan_bindings                            vlan_stats                             

    ------------------------------------ ------------ ----------- ------ ----------------- ---------------------------------------- ----------------------------------------

    070e59bc-ed70-4965-9519-dbec59b31033 {}           ""          "eth3" []                {1=128cf0c6-dfc7-47ce-8d4d-cc29b672ba34} {1=ef1dc78d-ac5c-4804-b3e9-649f2d435e9c}



    Physical_Switch table

    _uuid                                description         management_ips name    ports                                  switch_fault_status tunnel_ips  tunnels                              

    ------------------------------------ ------------------- -------------- ------- -------------------------------------- ------------------- ----------- --------------------------------------

    da9f2821-09f9-4c2f-beb9-4174de6fdd3b "OVS VTEP Emulator" []             "vtep0" [070e59bc-ed70-4965-9519-dbec59b31033] []                  [""] [58c2e509-f2ea-41e5-b12d-7a7c1d22a8eb]



    Tunnel table

    _uuid                                bfd_config_local                                            bfd_config_remote bfd_params bfd_status        local                                remote                             

    ------------------------------------ ----------------------------------------------------------- ----------------- ---------- ----------------- ------------------------------------ ------------------------------------

    58c2e509-f2ea-41e5-b12d-7a7c1d22a8eb {bfd_dst_ip="", bfd_dst_mac="00:23:20:00:00:01"} {}                {}         {enabled="false"} e5b3826b-0005-4cfa-8bac-e9f693cc3715 82bcef2f-1a8e-4191-b4b2-3255fdd6849c



    Ucast_Macs_Local table

    MAC                 _uuid                                ipaddr locator                              logical_switch                     

    ------------------- ------------------------------------ ------ ------------------------------------ ------------------------------------

    "7c:fe:90:29:23:36" b5155023-cf6f-45ff-a2e0-3ebba9d5db8b ""     e5b3826b-0005-4cfa-8bac-e9f693cc3715 128cf0c6-dfc7-47ce-8d4d-cc29b672ba34



    Ucast_Macs_Remote table

    MAC                 _uuid                                ipaddr              locator                              logical_switch                     

    ------------------- ------------------------------------ ------------------- ------------------------------------ ------------------------------------

    "fa:16:3e:26:96:a7" 91618bd9-3662-45d0-978c-97ed6659b8e6 "fdaa:e376:52b7::1" 82bcef2f-1a8e-4191-b4b2-3255fdd6849c 128cf0c6-dfc7-47ce-8d4d-cc29b672ba34

    "fa:16:3e:77:89:90" 5878f85d-9ddc-428d-84f0-87cc34d0e2cb ""          82bcef2f-1a8e-4191-b4b2-3255fdd6849c 128cf0c6-dfc7-47ce-8d4d-cc29b672ba34

    "fa:16:3e:57:32:99" 5878f85d-9ddc-428d-84f0-87cc34d0e2cb ""          82bcef2f-1a8e-4191-b4b2-3255fdd6849c 128cf0c6-dfc7-47ce-8d4d-cc29b672ba34

    "fa:16:3e:79:d2:11" 01e25a76-b403-430e-bdf1-b33c8bdf2bee ""          82bcef2f-1a8e-4191-b4b2-3255fdd6849c 128cf0c6-dfc7-47ce-8d4d-cc29b672ba34

    "fa:16:3e:94:66:0d" 891cf889-ac79-4dac-99b4-e6ec6611a988 ""          82bcef2f-1a8e-4191-b4b2-3255fdd6849c 128cf0c6-dfc7-47ce-8d4d-cc29b672ba34


    Get the switch running-config

    switch (config) # enable
    switch (config) # configure terminal
    switch (config) # show running-config


    The expected configuration should look like this:


    ## L3 configuration


       ip routing vrf default

       interface ethernet 1/4 no switchport force

       interface loopback 1

       interface ethernet 1/4 ip address

       interface loopback 1 ip address



    ## NVE configurations


       protocol nve

       interface nve 1

       interface nve 1 vxlan source interface loopback 1

       interface ethernet 1/7 nve mode only force

       ovs ovsdb server

       ovs ovsdb server listen tcp


    And finally we check if ping is available from the bare metal to the VMs and vice versa.



    1. Vlan number 0 and 1 cannot be assigned to the bare-metal ports, this limitation is caused from the ARP responder implementation.