This post provides quick overview of the Mellanox Poll Mode Driver (PMD) as a part of Data Plane Development Kit (DPDK).
This post is for developers who wish to use the DPDK API with Mellanox ConnectX-3 Pro, ConnectX-4 and ConnectX-5 adapter families.
- Mellanox Ethernet Driver
DPDK is a set of libraries and drivers for fast packet processing in user space. It provides a framework and common API for high speed networking applications.
For more information, refer to DPDK web site.
Poll Mode Driver (PMD) is designed for fast packet processing and low latency as follows:
Bypass kernel in receiving and sending packets
Avoid overhead of interrupt processing
mlx4 is the DPDK PMD for Mellanox ConnectX-3 Pro adapters. mlx4 is included starting from DPDK 2.0.
mlx5 is the DPDK PMD for Mellanox ConnectX-4/ConnectX-4 Lx/ConnectX-5 adapters. mlx5 is included starting from DPDK 2.2.
Mellanox PMD supports bare-metal, KVM and VMware.
Mellanox PMD is supported on x86_64, ARM and PPC.
For more details, please refer to Mellanox DPDK.
Value and Performance
Mellanox NICs with DPDK provide a secure and cost-effective solution.
Mellanox's implementation utilizes hardware based memory protection and translation providing secured isolation per process.
Mellanox's PMD implementation allows concurrent use of DPDK and non-DPDK processes on the same NIC providing a cost-effective solution with a single NIC.
Mellanox's ConnectX-5 NIC reaches 137Mpps packet frame rate (64Byte frame size) on a single port of 100GbE. See the demo here.
Mellanox PMD directly accesses HW for fast data-path processing while control-path still goes through kernel modules. Unlike other PMDs, Mellanox PMD does not rely on UIO/VFIO driver for control-path. Instead, control-path is built on top of libibverbs using the Raw Ethernet Accelerated Verbs API. Thus, the PCIe device is not required to be unbound from kernel network stack and standard Linux commands (e.g ethtool/ifconfig) can be run on the kernel device to query/update MTU and flow control parameters or to monitor statistics. This allows the PMD to coexist with kernel driver interface which still remains functional. Ports that are not being used by DPDK can send and receive traffic through the kernel network stack.
For security reasons, the PMD only deals with virtual memory addresses. The resource allocation is handled by the kernel complying with hardware specifications and it allows HW to handle virtual memory addresses directly ensuring that DPDK applications cannot access physical memory randomly (or memory that does not belong to the process).
libmlx4 / libmlx5 are the Mellanox user space drivers for Mellanox NICs.
mlx4_ib / mlx5_ib and mlx4_core / mlx5_core kernel modules are used for control path.
mlx4_en / mlx5_en is needed for bringing up the interfaces.
In addition to the upstream versions in dpdk.org, Mellanox releases LTS(Long-Term Support) version which is called MLNX_DPDK. MLNX_DPDK package branches off from a community release.
This release can be found in Mellanox DPDK and it is highly recommended to refer to the Quick Start Guide at the same page to install the release and achieve optimized performance. Release Note is also available.
DPDK community release
There are two versions available in the DPDK community - major and stable. Documents in the community are kept up-to-date - mlx5 and mlx4. The Quick Start Guide for MLNX_DPDK is mostly applicable to the community release, especially for installation and performance tuning.