Mellanox DPDK

Version 35

    This post provides quick overview of the Mellanox Poll Mode Driver (PMD) as a part of Data Plane Development Kit (DPDK).

    This post is for developers who wish to use the DPDK API with Mellanox ConnectX-3 Pro, ConnectX-4 and ConnectX-5 adapter families.

     

     

    References

     

     

    Overview

    DPDK is a set of libraries and drivers for fast packet processing in user space. It provides a framework and common API for high speed networking applications.

    For more information, refer to DPDK web site.

     

    Poll Mode Driver (PMD) is designed for fast packet processing and low latency as follows:

    • Bypass kernel in receiving and sending packets

    • Avoid overhead of interrupt processing

     

    mlx4 is the DPDK PMD for Mellanox ConnectX-3 Pro adapters. mlx4 is included starting from DPDK 2.0.

    mlx5 is the DPDK PMD for Mellanox ConnectX-4/ConnectX-4 Lx/ConnectX-5 adapters. mlx5 is included starting from DPDK 2.2.

    Both PMDs requires installing Mellanox OFED or Mellanox Ethernet Driver.

     

    Mellanox PMD supports bare-metal, KVM, Hyper-V and VMware.

    Mellanox PMD is supported on x86_64, ARM and PPC.

    For more details, please refer to Mellanox DPDK.

     

    Value and Performance

    Mellanox NICs with DPDK provide a secure and cost-effective solution.

    Mellanox's implementation utilizes hardware based memory protection and translation providing secured isolation per process.

    Mellanox's PMD implementation allows concurrent use of DPDK and non-DPDK processes on the same NIC providing a cost-effective solution with a single NIC.

     

    Mellanox's ConnectX-5 NIC reaches 137Mpps packet frame rate (64Byte frame size) on a single port of 100GbE. See the demo here.

     

    Architecture

    Mellanox PMD directly accesses HW for fast data-path processing while control-path still goes through kernel modules. Unlike other PMDs, Mellanox PMD does not rely on UIO/VFIO driver for control-path. Instead, control-path is built on top of libibverbs using the Raw Ethernet Accelerated Verbs API. Thus, the PCIe device is not required to be unbound from kernel network stack and standard Linux commands (e.g ethtool/ifconfig) can be run on the kernel device to query/update MTU and flow control parameters or to monitor statistics. This allows the PMD to coexist with kernel driver interface which still remains functional. Ports that are not being used by DPDK can send and receive traffic through the kernel network stack.

     

    For security reasons, the PMD only deals with virtual memory addresses. The resource allocation is handled by the kernel complying with hardware specifications and it allows HW to handle virtual memory addresses directly ensuring that DPDK applications cannot access physical memory randomly (or memory that does not belong to the process).

     

    libmlx4 / libmlx5 are the Mellanox user space drivers for Mellanox NICs.

    mlx4_ib / mlx5_ib and mlx4_core / mlx5_core kernel modules are used for control path.

    mlx4_en / mlx5_en is needed for bringing up the interfaces.

     

     

    Picture1.pngPicture1.png

     

    PMD Release

    In addition to the upstream versions in dpdk.org, Mellanox releases LTS(Long-Term Support) version which is called MLNX_DPDK. MLNX_DPDK package branches off from a community release.

     

    MLNX_DPDK release

    This release can be found in Mellanox DPDK and it is highly recommended to refer to the Quick Start Guide at the same page to install the release and achieve optimized performance. Release Note is also available.

     

    DPDK community release

    There are two versions available in the DPDK community - major and stable. Documents in the community are kept up-to-date - mlx5 and mlx4. The Quick Start Guide for MLNX_DPDK is mostly applicable to the community release, especially for installation and performance tuning.

     

    Note: Please install the recommended Mellanox OFED or Mellanox Ethernet Driver tested with the DPDK version you are running. The right version can be found in the Release Notes for MLNX_DPDK releases and on the dpdk.org community documentation for dpdk.org community releases. Below are the latest dpdk versions and their related driver and firmware versions:

     

    DPDK Version

    Network Adapter Cards/Firmware

    Driver version required

    Documents

    NIC Model

    Firmware version

    MLNX_DPDK 16.11_3.0

    ConnectX®-5 /
    ConnectX®-5 Ex

    v16.20.1010

    MLNX-OFED / MLNX_EN
    v4.1-1.0.0.x

    (Firmware is included with the driver download)

    ConnectX®-4 Lx

    v14.20.1010

    ConnectX®-4

    v12.20.1010

    ConnectX®-3 Pro

    v2.40.7000

    dpdk.org 17.05

    ConnectX®-5 /
    ConnectX®-5 Ex

    v16.19.1200

    MLNX-OFED / MLNX_EN

    v4.0-2.0.0.0

    (Firmware is included with the driver download)

    ConnectX®-4 Lx

    v14.18.2000

    ConnectX®-4

    v12.18.2000

    ConnectX®-3 Pro

    v2.40.5030