This post provides an example of how to configure VXLAN Tunnel Endpoint (VTEP) on Mellanox Spectrum Switches (MLNX-OS).
The reader is assumed to have experience in VMware NSX software.
VXLAN runs over the existing networking infrastructure and provides a means to “stretch” an L2 network. Only servers within the same VXLAN segment can communicate with each other. A network endpoint (such as Spectrum switch) that performs a translation from virtual (VMs) to physical network (bare metal servers) and back is called VXLAN Tunnel End-Point (VTEP). In virtual environments, it is typically required to use logical switches to forward traffic between different virtual machines (VMs) on the same physical host, between virtual machines and the physical machines and between networks. Virtual switch environments use an OVSDB management protocol for configuration and state discovery of the virtual networks. OVSDB protocol allows programmable access to the database of virtual switch configuration and performs the mapping between VNI on the virtual network to <PORT, VLAN> on the physical network.
The sample configuration includes the following:
- Two ESXi servers pre-configured with VXLAN networking using VMware NSX.
- Three network virtualization platform (NSX) controllers
- One Mellanox Spectrum switch connected to the ESXi servers and to a physical database server
- An out-of-band network for management and a VLAN (3) network to carry VXLAN traffic
1. Set MTU to Jumbo frames on the relevant switch ports (This should be aligned with the ESXi servers and database server):
switch (config) # interface ethernet 1/1-1/3 mtu 9216 force
2. Create VLAN 3 to carry VXLAN traffic
switch (config) # vlan 3
3. Set the switch interfaces towards the ESXi servers to be part of VLAN 3 in trunk mode.
switch (config) # interface ethernet 1 switchport mode trunk
switch (config) # interface ethernet 2 switchport mode trunk
4. Enable IP routing.
ip routing vrf default
5. Create an VLAN interface and assign it an IP address.
The IP address needs to be the default gateway of the "vxlan" netstack created by NSX after enabling VXLAN traffic on the hosts.
switch (config) # interface vlan 3
switch (config) # interface vlan 3 ip address 22.214.171.124 255.255.255.0
switch (config) # interface vlan 3 mtu 9216
Note: To check the default gateway in vSphere web client, select an ESXi host and go to Configure -> TCP/IP configuration.
6. Create a loopback interface. This interface will be the VTEP IP address assigned to this switch, which will communicate with the VTEPs on the ESXi servers by routing through "interface vlan 3".
switch (config) # interface loopback 1
switch (config) # interface loopback 1 ip address 126.96.36.199 255.255.255.255
7. Enable the Network Virtualization Edge (NVE) protocol, create an NVE interface, and specify that it is the source interface (loopback 1). The following commands enable VTEP on the switch and assign loopback1 as the interface to handle all Tx/Rx VXLAN traffic.
switch (config) # protocol nve
switch (config) # interface nve 1
switch (config) # interface nve 1 vxlan source interface loopback 1
8. Start the ovsdb server and connect it to the NSX controllers.
switch (config) # ovs ovsdb server
switch (config) # ovs ovsdb manager remote ssl ip address 10.130.200.100
switch (config) # ovs ovsdb manager remote ssl ip address 10.144.200.101
switch (config) # ovs ovsdb manager remote ssl ip address 10.144.200.102
9. Configure the port facing the database server as an NVE port.
switch (config) # interface ethernet 1/3 nve mode only force
Note: the switchport mode is controlled via the Controller and cannot be configured manually one the port is in nve mode.
10. Obtain the switch certificate. You will need this certificate when you configure NSX Manager later.
switch (config) # show crypto certificate name system-self-signed public-pem
Copy the certificate starting with the line:
until the line:
Make sure to include both of those lines.
NSX Controller Configuration
Add the Spectrum switch
This section describes how to configure the NSX Controller to add the Spectrum switch.
1. Add hosts to the replication cluster. Go to "Service Definitions" and select "Hardware Devices", Under "Replication Cluster", click Edit. Add the two ESXi servers to the replication cluster.
Note: All hosts you add to the replication cluster can replicate Broadcast, Unknown unicast and Multicast (BUM) traffic to other ESXi servers. When the switch needs to send BUM traffic to a virtual machine, it will select one of the hosts in the replication cluster and send the traffic to it. The host will then replicate it to all other ESXi hosts.
Note: It is recommended that you add at least two ESXi servers to the replication cluster for redundancy.
2. Add the Mellanox Spectrum switch to NSX.
Under the Hardware Devices tab click the plus sign to add a new hardware device. Enter a name for the new hardware device. Enter the switch certificate obtained earlier, and click OK.
Wait until the new switch is showing as "UP" under the Connectivity column. You might need to refresh the vSphere client a few times.
Map a Logical Switch to a Physical Switch Port
1. In NSX Manager select "Logical Switches". Right click the logical switch which you want to map to the physical switch port and select "Manage Hardware Bindings".
Note: The segment ID is the VNI.
2. Click the plus sign to add a new mapping. Click Select under the port column and select port "eth3". This corresponds to “interface ethernet 1/3" you configured earlier as an NVE port in the switch. Under the VLAN column, set the vlan that will map this logical switch to this specific switch port. You can have multiple logical switches mapped to the same port on a different vlan (for example to connect a firewall appliance to logical switches).
Note: For an "access" (untagged) configuration use VLAN 1 and click OK. In our case, we have no VLAN configured on the database server, therefore VLAN 1.
1. Configure the required VLAN on the database server and assign an IP address to it in the same subnet as other virtual machines in the logical switch.
2. Ping between a virtual machine connected to the logical switch and the database server.
3. On the switch, run:
switch (config) # show interfaces nve 1 mac-address-table
NVE Interface Logical Switch VNI ID Mac Address Address Type Remote Endpoint IP Address
------------- -------------- ------ ----------- ------------ --------------------------
1 240ac937-1ec2-371a-a 5002 00:50:56:99:33:09 remote configured 188.8.131.52
1 240ac937-1ec2-371a-a 5002 00:50:56:99:98:14 remote configured 184.108.40.206
You should see the MAC addresses of all the virtual machines in the connected logical switch and the MAC address of the database server.
4. Check the NVE Interface for status and counters.
switch (config) # show interfaces nve
Remote Manager IP Address Port Connection Type
------------------------- ---- ---------------
10.130.200.100 6640 ssl
10.144.200.101 6640 ssl
10.144.200.102 6640 ssl
Interface NVE 1 status:
Admin state: up
Source interface: loopback 1
NVE member interfaces: Eth1/1
0 encapsulated (Tx) NVE packets
0 decapsulated (Rx) NVE packets
0 dropped NVE-encapsulated packets
0 NVE-encapsulated packets with errors
5. Get the peer list per VNI.
# show interfaces nve 1 peers
NVE Interface Logical Switch VNI ID Peer IP Address
------------- -------------- ------ ----------------
1 415585bd-389b-3965-9 5002 220.127.116.11
1 415585bd-389b-3965-9 5002 18.104.22.168
Refer to MLNX-OS User Manual for full list of commands.