This post describes the procedure of installing DPDK 1.7.x and 1.8.x on bare metal Linux server with Mellanox ConnectX-3/ConnectX-3 Pro adapters and optimized libibverbs and libmlx4.
For DPDK 2.0 and above Mellanox PMD is part of the DPDK package. For more details please refer to Mellanox Poll Mode Driver (PMD) for DPDK.
1. Install MLNX_OFED 3.0-x.x.x. It can be downloaded from Mellanox site http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux
To install it, use the mlnxofedinstall installation script that is part of the package. for more advance option please refer to the MLNX_OFED user manual.
Note: MLNX_OFED installation will automatically upgrade the firmware unless specifically requested otherwise. in case firmware upgrade was done, a reboot is required, if not, only need to restart the openibd service
# service openibd restart
2. Verify that the MLNX_OFED was installed and ConnectX-3/ConnectX-3 Pro firmware is 2.34.5000 or higher
# ofed_info -s
CA type: MT4103
Number of ports: 2
Firmware version: 2.34.5000
Hardware version: 0
Node GUID: 0xf45214030029f470
System image GUID: 0xf45214030029f473.....
3. Make sure all port are set to Ethernet (port type). Refer to HowTo Change Port Type in Mellanox ConnectX-3 Adapter
4. Download and build DPDK version 1.8 (it will work the same way with version 1.7) from DPDK download
# wget dpdk.org/browse/dpdk/snapshot/dpdk-1.8.0.tar.gz
# tar -zxvf dpdk-1.8.0.tar.gz
# cd dpdk-1.8.0
# make install T=x86_64-native-linuxapp-gcc
5. Download and compile Mellanox optimized libraries from Mellanox PMD for DPDK
# wget www.mellanox.com/downloads/Drivers/mlx4_pmd_v1.7-8_2.8.4.tar.gz
# tar -zxvf mlx4_pmd_v1.7-8_2.8.4.tar.gz
6. Run the compilation script - ./compile_mlx4_pmd.sh -s <RTE_SDK> -t <RTE_TARGET>
# cd mlx4_pmd_v1.7-8_2.8.4
# ./compile_mlx4_pmd.sh -s /var/soft/dpdk/dpdk-1.8/dpdk-1.8.0 -t x86_64-native-linuxapp-gcc
The output of the PMD compilation is a shared object, librte_pmd_mlx4.so is created under the mlx4_pmd_v1.7-8_2.8.4/librte_pmd_mlx4/librte_pmd_mlx4.so.
The librte_pmd_mlx4.so should be dynamically loaded during the DPDK application run with the –d flag (see example below)
7. Find out the NUMA node close to the network device (port) you want to use for running DPDK. The NUMA node can be retrieved by - cat /sys/class/net/[interface]/device/numa_node command.
mlx4_0 port 1 ==> eth4 (Up)
mlx4_0 port 2 ==> eth5 (Up)
# cat /sys/class/net/eth4/device/numa_node
8. Run the configuration script - ./configure_mlx4_pmd.sh -s #NUMA-node
# ./configure_mlx4_pmd.sh -s 1
2048 pages with size 2048 will be configured
Please run /etc/init.d/openibd restart
9. In case VLAN filering is used run ./configure_mlx4_pmd.sh -s #NUMA-node -v. This might lead to performance degradation. The configuration script configure the hugepages and allocate them on the #NUMA-node, the script also configure the flow steering parameters (num_log_entry) in mlnx driver.
10. Restart MLNX_OFED
# service openibd restart
11. Find out the cores on the relevant NUMA nodes (1 in our case)
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
On-line CPU(s) list: 0-19
Thread(s) per core: 1
Core(s) per socket: 10
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
CPU MHz: 2800.158
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 25600K
NUMA node0 CPU(s): 0-9
NUMA node1 CPU(s): 10-19
It means we need to use core 10 and higher for our DPDK application. a good choice for the -c parameter her would be 0x1C00.
Note: the DPDK require additional core for DPDK itself.
12. Find out the pci slot of Mellanox NIC:
# lspci | grep Mell
86:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
Please note - even for a dual port NIC there is only one PCI address.
13. Verify installation by simple run of the testpmd application with two cores (no RSS)
# dpdk-1.8.0/x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd -m 2048 -w 86:00.0 -c 0x1C00 -n 4 -d /var/soft/pmd/mlx4_pmd_v1.7-8_2.8.4/librte_pmd_mlx4/librte_pmd_mlx4.so -- --numa --burst=64 --rxd=4096 --txd=4096 --mbcache=512 --rxq=1 --txq=1 --total-num-mbufs=40480 --forward-mode=io --nb-cores=2 --mbuf-size=2048 -a -i
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 8 on socket 0
EAL: Detected lcore 6 as core 9 on socket 0
EAL: Detected lcore 7 as core 10 on socket 0
EAL: Detected lcore 8 as core 11 on socket 0
EAL: Detected lcore 9 as core 12 on socket 0
EAL: Detected lcore 10 as core 0 on socket 1
EAL: Detected lcore 11 as core 1 on socket 1
EAL: Detected lcore 12 as core 2 on socket 1
EAL: Detected lcore 13 as core 3 on socket 1
EAL: Detected lcore 14 as core 4 on socket 1
EAL: Detected lcore 15 as core 8 on socket 1
EAL: Detected lcore 16 as core 9 on socket 1
EAL: Detected lcore 17 as core 10 on socket 1
EAL: Detected lcore 18 as core 11 on socket 1
EAL: Detected lcore 19 as core 12 on socket 1
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 20 lcore(s)
EAL: Setting up memory...
EAL: Ask a virtual area of 0x100000000 bytes
EAL: Virtual area found at 0x7fa32f200000 (size = 0x100000000)
EAL: Requesting 1024 pages of size 2MB from socket 1
EAL: TSC frequency is ~2800003 KHz
EAL: open shared lib /var/soft/pmd/mlx4_pmd_v1.7-8_2.8.4/librte_pmd_mlx4/librte_pmd_mlx4.so
EAL: Master core 10 is ready (tid=2fbe8800)
PMD: ENICPMD trace: rte_enic_pmd_init
EAL: Core 12 is ready (tid=2e581700)
EAL: Core 11 is ready (tid=2ef82700)
EAL: PCI device 0000:86:00.0 on NUMA socket 1
EAL: probe driver: 15b3:1007 librte_pmd_mlx4
libibverbs: Warning: couldn't open config directory '/usr/local/etc/libibverbs.d'.
PMD: librte_pmd_mlx4: PCI information matches, using device "mlx4_0" (VF: false)
PMD: librte_pmd_mlx4: 2 port(s) detected
PMD: librte_pmd_mlx4: port 1 MAC address is f4:52:14:29:f4:71
PMD: librte_pmd_mlx4: port 2 MAC address is f4:52:14:29:f4:72
Set io packet forwarding mode
Configuring Port 0 (socket 1)
PMD: librte_pmd_mlx4: 0x833400: TX queues number update: 0 -> 1
PMD: librte_pmd_mlx4: 0x833400: RX queues number update: 0 -> 1
Port 0: F4:52:14:29:F4:71
Configuring Port 1 (socket 1)
PMD: librte_pmd_mlx4: 0x833440: TX queues number update: 0 -> 1
PMD: librte_pmd_mlx4: 0x833440: RX queues number update: 0 -> 1
Port 1: F4:52:14:29:F4:72
Checking link statuses...
Port 0 Link Up - speed 40000 Mbps - full-duplex
Port 1 Link Up - speed 40000 Mbps - full-duplex
Start automatic packet forwarding
io packet forwarding - CRC stripping disabled - packets/burst=64
nb forwarding cores=2 - nb forwarding ports=2
RX queues=1 - RX desc=4096 - RX free threshold=32
RX threshold registers: pthresh=8 hthresh=8 wthresh=0
TX queues=1 - TX desc=4096 - TX free threshold=0
TX threshold registers: pthresh=32 hthresh=0 wthresh=0
TX RS bit threshold=0 - TXQ flags=0x0
14. Generate traffic and check the counters via show port stats all command.