HowTo Build and Run OVS-DPDK with Mellanox PMD

Version 6

    This article explains how to compile and run OVS-DPDK with Mellanox PMD. OVS-DPDK can run with Mellanox ConnectX-3 and ConnectX-4 network adapters. The datapath of OVS was implemented in kernel but the OVS community has been putting huge effort to accelerate the datapath with DPDK.

     

     

    References

     

    Supported Versions of OVS and DPDK

    The most recent OVS releases are 2.5.1 and 2.6.1. According to INSTALL.DPDK.md included in OVS releases, OVS 2.5.1 requires MLNX_DPDK 2.2 and the current branch-2.6 requires the latest DPDK 16.07. The two combinations will be included in this post.

     

    1. OVS 2.5.1 + MLNX_DPDK 2.2_4.2

    DPDK 2.2 is a LTS (long-term support) version for Mellanox PMD. Mellanox will continue to release separate versions (2.2_x.x) based on DPDK 2.2 in mellanox.com. As of 4 Nov 2016, 2.2_4.2 is the latest release.

     

    2. OVS 2.6.1 + DPDK 16.07

    Since DPDK 16.07, Mellanox actively aligns with community release schedule. There's no need to download separate packages from 16.07. DPDK 16.04 isn't recommended.

     

    How to Build OVS + DPDK

    Please refer to the Quick Start Guide of Mellanox PMD and the INSTALL.DPDK.md in an OVS release for details. Please note that libibverbs should be linked using LDFLAGS=-libverbs  flag when building OVS, because Mellanox PMD uses the library.

    The followings are brief summary of the commands to build DPDK and OVS.

     

    OVS 2.5.1 with MLNX_DPDK 2.2_4.2

    OVS 2.5.1 requires DPDK being compiled as a combined library, this can be enabled by setting CONFIG_RTE_BUILD_COMBINE_LIBS=y in config/common_linuxapp.

    Note: There is a minor bug in OVS 2.5.1, the link speed definitions are changed in DPDK, but the OVS does not reflect the change. It was fixed with OVS supporting DPDK 16.04. The attached patch (fix_dpdk_link_speed_definitions.patch) is needed to make it work with DPDK 2.2.

     

    1. DPDK 2.2_4.2

    # wget http://www.mellanox.com/downloads/Drivers/MLNX_DPDK_2.2_4.2.tar.gz

    # tar xf MLNX_DPDK_2.2_4.2.tar.gz

    # cd MLNX_DPDK_2.2_4.2

    # sed -i 's/\(CONFIG_RTE_BUILD_COMBINE_LIBS=\)n/\1y/g' config/common_linuxapp

    # make -j install T=x86_64-native-linuxapp-gcc DESTDIR=install

    # export DPDK_DIR=$PWD

    # export DPDK_BUILD=$DPDK_DIR/x86_64-native-linuxapp-gcc

     

    2. OVS 2.5.1

    # wget http://openvswitch.org/releases/openvswitch-2.5.1.tar.gz

    # tar xf openvswitch-2.5.1.tar.gz

    # cd openvswitch-2.5.1

    # patch -p1 < fix_dpdk_link_speed_definitions.patch

    # ./boot.sh

    # ./configure --with-dpdk=$DPDK_BUILD

    # make -j LDFLAGS=-libverbs

    # make install

    OVS 2.6.1 with DPDK 16.07

    1. DPDK 16.07

    # wget http://fast.dpdk.org/rel/dpdk-16.07.tar.xz

    # tar xf dpdk-16.07.tar.xz

    # cd dpdk-16.07

    # sed -i 's/\(CONFIG_RTE_LIBRTE_MLX5_PMD=\)n/\1y/g' config/common_base

    # sed -i 's/\(CONFIG_RTE_LIBRTE_MLX4_PMD=\)n/\1y/g' config/common_base

    # sed -i 's/\(CONFIG_RTE_LIBRTE_MLX4_SGE_WR_N=\)4/\11/g' config/common_base

    # make -j install T=x86_64-native-linuxapp-gcc DESTDIR=install

    # export DPDK_DIR=$PWD

    # export DPDK_BUILD=$DPDK_DIR/x86_64-native-linuxapp-gcc

    2. OVS/branch-2.6.1

    # wget http://openvswitch.org/releases/openvswitch-2.6.1.tar.gz

    # tar xf openvswitch-2.6.1.tar.gz

    # cd openvswitch-2.6.1

    # ./boot.sh

    # ./configure --with-dpdk=$DPDK_BUILD

    # make -j LDFLAGS=-libverbs

    # make install

     

     

    Knows Issues

    It is recommended to follow instructions in INSTALL.DPDK.md of OVS releases when running the OVS-DPSK, there are a few caveats when running OVS-DPDK with Mellanox PMD.

     

    1. Binding to user-level driver

    Unlike other NICs, Mellanox PMD does not need unbinding from the kernel driver and binding to an user-level driver. By default, when a DPDK app runs, every Mellanox adapters are probed and attached to the app. In order to attach only specific adapters, white-lists (-w) or black-lists (-b) can be added to EAL options as usual.

     

    2. "Cannot allocate memory" error

    OVS 2.5.0 may fail to run with Mellanox PMD. The error is related to a security fix introduced lately in OFED 3.3 which causes queue allocation failure. The patch prevents calling write() from a forked process. If ovs-vswitchd is run with '--detach' option, it initializes DPDK PMD in a parent process and then tries to create/configure queues in a fork'd child process. That is why the system call-to-allocate resources failes and returned "Cannot allocate memory". A simple solution is to run ovs-vswitchd without '--detach' option, or, the attached patch (move_dpdk_init_after_fork.patch) could fix the problem. This simple patch is to call dpdk_init() after fork().

    This issue hasalready been fixed in OVS 2.6.0. OVS community code was changed to initialize PMD after fork(). PMD is initialized and configured in a child process and DPDK is no longer initialized before deamonize_start(). The change also affects "delays dpdk initialization until after the OVS database connection is established, at which point ovs initializes librte".

     

     

    Performance Optimization

    It depends on the testing environment, so it is highly recommended to refer to the Quick Start Guide. There are many options to control. For example, to get high throughput, the followings could be helpful in general :

    • Disable flow control (pause frame)
    • Set PCIe MaxReadRequest to 1024
    • Enable txq_inline
    • Disable IO non-posted prefetch in BIOS
    • Generate multiple flows when using multiple queues