HowTo Install DPDK 1.7/1.8 with ConnectX-3 Adapter

Version 6

    This post describes the procedure of installing DPDK 1.7.x and 1.8.x on bare metal Linux server with Mellanox ConnectX-3/ConnectX-3 Pro adapters and optimized libibverbs and libmlx4.

    For DPDK 2.0 and above Mellanox PMD is part of the DPDK package. For more details please refer to Mellanox Poll Mode Driver (PMD) for DPDK.

     

    References

     

    Configuration

     

    1. Install MLNX_OFED 3.0-x.x.x. It can be downloaded from Mellanox site http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux

    To install it, use the mlnxofedinstall installation script that is part of the package. for more advance option please refer to the MLNX_OFED user manual.

     

    Note: MLNX_OFED installation will automatically upgrade the firmware unless specifically requested otherwise. in case firmware upgrade was done, a reboot is required, if not, only need to restart the openibd service

    # service openibd restart

     

    2. Verify that the MLNX_OFED was installed and ConnectX-3/ConnectX-3 Pro firmware is 2.34.5000 or higher

    # ofed_info -s

    MLNX_OFED_LINUX-3.0-1.0.1:

     

    # ibstat

    CA 'mlx4_0'

            CA type: MT4103

            Number of ports: 2

            Firmware version: 2.34.5000

            Hardware version: 0

            Node GUID: 0xf45214030029f470

            System image GUID: 0xf45214030029f473

         .....
         .....    

    3. Make sure all port are set to Ethernet (port type). Refer to HowTo Change Port Type in Mellanox ConnectX-3 Adapter


    4. Download and build DPDK version 1.8  (it will work the same way with version 1.7) from DPDK download

    # wget dpdk.org/browse/dpdk/snapshot/dpdk-1.8.0.tar.gz

    # tar -zxvf dpdk-1.8.0.tar.gz

    # cd dpdk-1.8.0

    # make install T=x86_64-native-linuxapp-gcc

     

    5. Download and compile Mellanox optimized libraries from Mellanox PMD for DPDK

    # wget www.mellanox.com/downloads/Drivers/mlx4_pmd_v1.7-8_2.8.4.tar.gz

    # tar -zxvf mlx4_pmd_v1.7-8_2.8.4.tar.gz

     

    6. Run the compilation script - ./compile_mlx4_pmd.sh -s <RTE_SDK> -t <RTE_TARGET>

    # cd mlx4_pmd_v1.7-8_2.8.4

    # ./compile_mlx4_pmd.sh -s /var/soft/dpdk/dpdk-1.8/dpdk-1.8.0 -t x86_64-native-linuxapp-gcc

    The output of the PMD compilation is a shared object, librte_pmd_mlx4.so is created under the mlx4_pmd_v1.7-8_2.8.4/librte_pmd_mlx4/librte_pmd_mlx4.so.

    The librte_pmd_mlx4.so should be dynamically loaded during the DPDK application run with the –d flag (see example below)

     

    7. Find out the NUMA node close to the network device (port) you want to use for running DPDK. The NUMA node can be retrieved by  - cat /sys/class/net/[interface]/device/numa_node command.

    # ibdev2netdev

    mlx4_0 port 1 ==> eth4 (Up)

    mlx4_0 port 2 ==> eth5 (Up)

     

    # cat /sys/class/net/eth4/device/numa_node

    1

     

    8. Run the configuration script - ./configure_mlx4_pmd.sh -s #NUMA-node

    # ./configure_mlx4_pmd.sh -s 1

    2048 pages with size 2048 will be configured

    Please run /etc/init.d/openibd restart

     

    9. In case VLAN filering is used run ./configure_mlx4_pmd.sh -s #NUMA-node -v.  This might lead to performance degradation. The configuration script configure the hugepages and allocate them on the #NUMA-node, the script also configure the flow steering parameters (num_log_entry) in mlnx driver.

     

    10. Restart MLNX_OFED

    # service openibd restart

     

    11. Find out the cores on the relevant NUMA nodes (1 in our case)

    # lscpu

    Architecture:          x86_64

    CPU op-mode(s):        32-bit, 64-bit

    Byte Order:            Little Endian

    CPU(s):                20

    On-line CPU(s) list:   0-19

    Thread(s) per core:    1

    Core(s) per socket:    10

    Socket(s):             2

    NUMA node(s):          2

    Vendor ID:             GenuineIntel

    CPU family:            6

    Model:                 62

    Stepping:              4

    CPU MHz:               2800.158

    BogoMIPS:              5599.18

    Virtualization:        VT-x

    L1d cache:             32K

    L1i cache:             32K

    L2 cache:              256K

    L3 cache:              25600K

    NUMA node0 CPU(s):     0-9

    NUMA node1 CPU(s):     10-19

    It means we need to use core 10 and higher for our DPDK application. a good choice for the -c parameter her would be 0x1C00.

    Note: the DPDK require additional core for DPDK itself.

     

    12. Find out the pci slot of Mellanox NIC:

    # lspci | grep Mell

    86:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

    Please note - even for a dual port NIC there is only one PCI address.

     

    13. Verify installation by simple run of the testpmd application with two cores (no RSS)

    # dpdk-1.8.0/x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd -m 2048 -w 86:00.0 -c 0x1C00 -n 4 -d /var/soft/pmd/mlx4_pmd_v1.7-8_2.8.4/librte_pmd_mlx4/librte_pmd_mlx4.so -- --numa --burst=64 --rxd=4096 --txd=4096 --mbcache=512  --rxq=1 --txq=1 --total-num-mbufs=40480 --forward-mode=io --nb-cores=2 --mbuf-size=2048 -a -i

    EAL: Detected lcore 0 as core 0 on socket 0

    EAL: Detected lcore 1 as core 1 on socket 0

    EAL: Detected lcore 2 as core 2 on socket 0

    EAL: Detected lcore 3 as core 3 on socket 0

    EAL: Detected lcore 4 as core 4 on socket 0

    EAL: Detected lcore 5 as core 8 on socket 0

    EAL: Detected lcore 6 as core 9 on socket 0

    EAL: Detected lcore 7 as core 10 on socket 0

    EAL: Detected lcore 8 as core 11 on socket 0

    EAL: Detected lcore 9 as core 12 on socket 0

    EAL: Detected lcore 10 as core 0 on socket 1

    EAL: Detected lcore 11 as core 1 on socket 1

    EAL: Detected lcore 12 as core 2 on socket 1

    EAL: Detected lcore 13 as core 3 on socket 1

    EAL: Detected lcore 14 as core 4 on socket 1

    EAL: Detected lcore 15 as core 8 on socket 1

    EAL: Detected lcore 16 as core 9 on socket 1

    EAL: Detected lcore 17 as core 10 on socket 1

    EAL: Detected lcore 18 as core 11 on socket 1

    EAL: Detected lcore 19 as core 12 on socket 1

    EAL: Support maximum 128 logical core(s) by configuration.

    EAL: Detected 20 lcore(s)

    EAL: Setting up memory...

    EAL: Ask a virtual area of 0x100000000 bytes

    EAL: Virtual area found at 0x7fa32f200000 (size = 0x100000000)

    EAL: Requesting 1024 pages of size 2MB from socket 1

    EAL: TSC frequency is ~2800003 KHz

    EAL: open shared lib /var/soft/pmd/mlx4_pmd_v1.7-8_2.8.4/librte_pmd_mlx4/librte_pmd_mlx4.so

    EAL: Master core 10 is ready (tid=2fbe8800)

    PMD: ENICPMD trace: rte_enic_pmd_init

    EAL: Core 12 is ready (tid=2e581700)

    EAL: Core 11 is ready (tid=2ef82700)

    EAL: PCI device 0000:86:00.0 on NUMA socket 1

    EAL:   probe driver: 15b3:1007 librte_pmd_mlx4

    libibverbs: Warning: couldn't open config directory '/usr/local/etc/libibverbs.d'.

    PMD: librte_pmd_mlx4: PCI information matches, using device "mlx4_0" (VF: false)

    PMD: librte_pmd_mlx4: 2 port(s) detected

    PMD: librte_pmd_mlx4: port 1 MAC address is f4:52:14:29:f4:71

    PMD: librte_pmd_mlx4: port 2 MAC address is f4:52:14:29:f4:72

    Set io packet forwarding mode

    Auto-start selected

    Interactive-mode selected

    Configuring Port 0 (socket 1)

    PMD: librte_pmd_mlx4: 0x833400: TX queues number update: 0 -> 1

    PMD: librte_pmd_mlx4: 0x833400: RX queues number update: 0 -> 1

    Port 0: F4:52:14:29:F4:71

    Configuring Port 1 (socket 1)

    PMD: librte_pmd_mlx4: 0x833440: TX queues number update: 0 -> 1

    PMD: librte_pmd_mlx4: 0x833440: RX queues number update: 0 -> 1

    Port 1: F4:52:14:29:F4:72

    Checking link statuses...

    Port 0 Link Up - speed 40000 Mbps - full-duplex

    Port 1 Link Up - speed 40000 Mbps - full-duplex

    Done

    Start automatic packet forwarding

      io packet forwarding - CRC stripping disabled - packets/burst=64

      nb forwarding cores=2 - nb forwarding ports=2

      RX queues=1 - RX desc=4096 - RX free threshold=32

      RX threshold registers: pthresh=8 hthresh=8 wthresh=0

      TX queues=1 - TX desc=4096 - TX free threshold=0

      TX threshold registers: pthresh=32 hthresh=0 wthresh=0

      TX RS bit threshold=0 - TXQ flags=0x0

    testpmd>

     

    14. Generate traffic and check the counters via show port stats all command.