HowTo Configure VTEP using VMware NSX on Mellanox Spectrum Switches (MLNX-OS)

Version 10

    This post provides an example of how to configure VXLAN Tunnel Endpoint (VTEP) on Mellanox Spectrum Switches (MLNX-OS).

    The reader is assumed to have experience in VMware NSX software.





    VXLAN runs over the existing networking infrastructure and provides a means to “stretch” an L2 network. Only servers within the same VXLAN segment can communicate with each other. A network endpoint (such as Spectrum switch) that performs a translation from virtual (VMs) to physical network (bare metal servers) and back is called VXLAN Tunnel End-Point (VTEP). In virtual environments, it is typically required to use logical switches to forward traffic between different virtual machines (VMs) on the same physical host, between virtual machines and the physical machines and between networks. Virtual switch environments use an OVSDB management protocol for configuration and state discovery of the virtual networks. OVSDB protocol allows programmable access to the database of virtual switch configuration and performs the mapping between VNI on the virtual network to <PORT, VLAN> on the physical network.



    The sample configuration includes the following:

    • Two ESXi servers pre-configured with VXLAN networking using VMware NSX.
    • Three network virtualization platform (NSX) controllers
    • One Mellanox Spectrum switch connected to the ESXi servers and to a physical database server
    • An out-of-band network for management and a VLAN (3) network to carry VXLAN traffic



    Switch Configuration

    1. Set MTU to Jumbo frames on the relevant switch ports (This should be aligned with the ESXi servers and database server):

    switch (config) # interface ethernet 1/1-1/3 mtu 9216 force


    2. Create VLAN 3 to carry VXLAN traffic

    switch (config) # vlan 3


    3. Set the switch interfaces towards the ESXi servers to be part of VLAN 3 in trunk mode.

    switch (config) # interface ethernet 1 switchport mode trunk
    switch (config) # interface ethernet 2 switchport mode trunk


    4. Enable IP routing.

    ip routing vrf default


    5. Create an VLAN interface and assign it an IP address.

    The IP address needs to be the default gateway of the "vxlan" netstack created by NSX after enabling VXLAN traffic on the hosts.

    switch (config) # interface vlan 3

    switch (config) # interface vlan 3 ip address

    switch (config) # interface vlan 3 mtu 9216


    Note: To check the default gateway in vSphere web client, select an ESXi host and go to Configure -> TCP/IP configuration.



    6. Create a loopback interface. This interface will be the VTEP IP address assigned to this switch, which will communicate with the VTEPs on the ESXi servers by routing through "interface vlan 3".

    switch (config) # interface loopback 1

    switch (config) # interface loopback 1 ip address


    7. Enable the Network Virtualization Edge (NVE) protocol, create an NVE interface, and specify that it is the source interface (loopback 1). The following commands enable VTEP on the switch and assign loopback1 as the interface to handle all Tx/Rx VXLAN traffic.

    switch (config) # protocol nve

    switch (config) # interface nve 1

    switch (config) # interface nve 1 vxlan source interface loopback 1


    8. Start the ovsdb server and connect it to the NSX controllers.

    switch (config) # ovs ovsdb server

    switch (config) # ovs ovsdb manager remote ssl ip address

    switch (config) # ovs ovsdb manager remote ssl ip address

    switch (config) # ovs ovsdb manager remote ssl ip address


    9. Configure the port facing the database server as an NVE port.

    switch (config) # interface ethernet 1/3 nve mode only force


    Note: the switchport mode is controlled via the Controller and cannot be configured manually one the port is in nve mode.


    10. Obtain the switch certificate. You will need this certificate when you configure NSX Manager later.

    switch (config) # show crypto certificate name system-self-signed public-pem


    Copy the certificate starting with the line:


    until the line:

    -----END CERTIFICATE-----

    Make sure to include both of those lines.


    NSX Controller Configuration

    Add the Spectrum switch

    This section describes how to configure the NSX Controller to add the Spectrum switch.

    1. Add hosts to the replication cluster.  Go to "Service Definitions" and select "Hardware Devices", Under "Replication Cluster", click Edit. Add the two ESXi servers to the replication cluster.

    Note: All hosts you add to the replication cluster can replicate Broadcast, Unknown unicast and Multicast (BUM) traffic to other ESXi servers. When the switch needs to send BUM traffic to a virtual machine, it will select one of the hosts in the replication cluster and send the traffic to it. The host will then replicate it to all other ESXi hosts.


    Note: It is recommended that you add at least two ESXi servers to the replication cluster for redundancy.



    2. Add the Mellanox Spectrum switch to NSX.

    Under the Hardware Devices tab click the plus sign to add a new hardware device. Enter a name for the new hardware device. Enter the switch certificate obtained earlier, and click OK.



    Wait until the new switch is showing as "UP" under the Connectivity column. You might need to refresh the vSphere client a few times.


    Map a Logical Switch to a Physical Switch Port


    1. In NSX Manager select "Logical Switches". Right click the logical switch which you want to map to the physical switch port and select "Manage Hardware Bindings".

    Note: The segment ID is the VNI.


    2. Click the plus sign to add a new mapping. Click Select under the port column and select port "eth3". This corresponds to “interface ethernet 1/3" you configured earlier as an NVE port in the switch. Under the VLAN column, set the vlan that will map this logical switch to this specific switch port. You can have multiple logical switches mapped to the same port on a different vlan (for example to connect a firewall appliance to logical switches).

    Note: For an "access"  (untagged) configuration use VLAN 1 and click OK. In our case, we have no VLAN configured on the database server, therefore VLAN 1.




    1. Configure the required VLAN on the database server and assign an IP address to it in the same subnet as other virtual machines in the logical switch.


    2. Ping between a virtual machine connected to the logical switch and the database server.


    3. On the switch, run:

    switch (config) # show interfaces nve 1 mac-address-table

    NVE Interface   Logical Switch         VNI ID     Mac Address         Address Type        Remote Endpoint IP Address

    -------------   --------------         ------     -----------         ------------        --------------------------

    1               240ac937-1ec2-371a-a   5002       00:50:56:99:33:09   remote configured

    1               240ac937-1ec2-371a-a   5002       00:50:56:99:98:14   remote configured

    You should see the MAC addresses of all the virtual machines in the connected logical switch and the MAC address of the database server.


    4. Check the NVE Interface for status and counters.

    switch  (config) # show interfaces nve


    Remote Manager IP Address                        Port    Connection Type

    -------------------------                        ----    ---------------                                   6640    ssl                                   6640    ssl                                   6640    ssl


    Interface NVE 1 status:

      Admin state: up

      Source interface: loopback 1

      NVE member interfaces:  Eth1/1


      0                    encapsulated (Tx) NVE packets

      0                    decapsulated (Rx) NVE packets

      0                    dropped NVE-encapsulated packets

      0                    NVE-encapsulated packets with errors



    5. Get the peer list per VNI.

    # show interfaces nve 1 peers


    NVE Interface   Logical Switch         VNI ID     Peer IP Address

    -------------   --------------         ------     ----------------

    1               415585bd-389b-3965-9   5002

    1               415585bd-389b-3965-9   5002


    Refer to MLNX-OS User Manual for full list of commands.