2 Replies Latest reply on Nov 15, 2018 4:46 AM by rghodgkin@pureport.com

    VXLAN offload/RSS not working under OVS switch scenario


      Hi folks,

             We are using two Mellanox Connectx-5 EN (2-port 100 GBE) cards to validate performance under an Openstack vxlan tenant scenario using OVS bridges.   The Mellanox cards are NOT in SR-IOV mode and are currently connected back-to-back to create the vxlan tenant network with core IP addresses applied directly to the ports.  It is our understanding from reading Mellanox literature that under this scenario the Mellanox should be offloading the VXLAN tunnel udp encap/decap somewhat automatically (as per the OVS switch 'vxlan' interface configuration); however, at the moment this does not seem to be occurring.  Traffic is being generated across multiple VNID's, however all ingress traffic from the vxlan/tenant network is getting processed by a single core softirq process.  We are not able to verify if this is b/c something is not setup properly in regard to the Mellanox card/drivers or if we do not have RSS tuned as needed.   Further details below, any assistance on configuration/debug suggestions or updated documentation for this scenario under Connectx-5 would be appreciated!  


      Further details:

      We have been reading from the following document although it is targeted for Connectx-3 ( HowTo Configure VXLAN for ConnectX-3 Pro (Linux OVS) .    This document shows several items to be configured (DMFS & VXLAN port number) but it seems both of these should be enabled now under Connectx-5 and the 'mlx5_core' driver.    Also the recommended debug/log steps in this document are no longer supported under Connectx-5.   In our setup we are using the apt package install for OVS. 




      Mellanox card:  Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]

      Server version:   Ubuntu 18.04  (kernel 4.15.0-36-generic)

      OVS version:   ovs-vsctl (Open vSwitch) 2.9.0



      root@us-ral-1a-e1:~# mlxfwmanager --query

      Querying Mellanox devices firmware ...



      Device #1:


        Device Type:      ConnectX5

        Part Number:      MCX516A-CCA_Ax

        Description:      ConnectX-5 EN network interface card; 100GbE dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6

        PSID:             MT_0000000012

        PCI Device Name:  /dev/mst/mt4119_pciconf0

        Base GUID:        98039b03004dd018

        Base MAC:         98039b4dd018

        Versions:         Current        Available

           FW             16.23.1020     N/A

           PXE            3.5.0504       N/A

           UEFI           14.16.0017     N/A


        Status:           No matching image found