We are using two Mellanox Connectx-5 EN (2-port 100 GBE) cards to validate performance under an Openstack vxlan tenant scenario using OVS bridges. The Mellanox cards are NOT in SR-IOV mode and are currently connected back-to-back to create the vxlan tenant network with core IP addresses applied directly to the ports. It is our understanding from reading Mellanox literature that under this scenario the Mellanox should be offloading the VXLAN tunnel udp encap/decap somewhat automatically (as per the OVS switch 'vxlan' interface configuration); however, at the moment this does not seem to be occurring. Traffic is being generated across multiple VNID's, however all ingress traffic from the vxlan/tenant network is getting processed by a single core softirq process. We are not able to verify if this is b/c something is not setup properly in regard to the Mellanox card/drivers or if we do not have RSS tuned as needed. Further details below, any assistance on configuration/debug suggestions or updated documentation for this scenario under Connectx-5 would be appreciated!
We have been reading from the following document although it is targeted for Connectx-3 ( HowTo Configure VXLAN for ConnectX-3 Pro (Linux OVS) . This document shows several items to be configured (DMFS & VXLAN port number) but it seems both of these should be enabled now under Connectx-5 and the 'mlx5_core' driver. Also the recommended debug/log steps in this document are no longer supported under Connectx-5. In our setup we are using the apt package install for OVS.
Mellanox card: Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
Server version: Ubuntu 18.04 (kernel 4.15.0-36-generic)
OVS version: ovs-vsctl (Open vSwitch) 2.9.0
root@us-ral-1a-e1:~# mlxfwmanager --query
Querying Mellanox devices firmware ...
Device Type: ConnectX5
Part Number: MCX516A-CCA_Ax
Description: ConnectX-5 EN network interface card; 100GbE dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6
PCI Device Name: /dev/mst/mt4119_pciconf0
Base GUID: 98039b03004dd018
Base MAC: 98039b4dd018
Versions: Current Available
FW 16.23.1020 N/A
PXE 3.5.0504 N/A
UEFI 14.16.0017 N/A
Status: No matching image found