HowTo Configure DPDK Packet Generator for ConnectX-4

Version 11

    Measuring packet forwarding throughput requires an expensive traffic generator. However, not every test case needs to utilize the full blown features of such generators, especially for 40GbE and 100GbE interfaces. In many cases, target applications can not reach line rate with 64-byte packets. If an expensive packet generator is not available, a good alternative exists based on DPDK - pktgen-dpdk.

     

    References

     

    Prerequisites

     

    Where to Get

    Tarballs are officially linked in dpdk.org:

     

    Or, it can be downloaded directly from a git repository:

    git clone git://dpdk.org/apps/pktgen-dpdk

     

    Documentation can be found here.

     

    Installation

    Refer to documentations of each package.

     

    To run pktgen-dpdk please refer to the following page:

     

    Simple Commands to Build pktgen-dpdk

    # Download pktgen from pktgen-dpdk - Traffic generator powered by DPDK

    wget http://dpdk.org/browse/apps/pktgen-dpdk/snapshot/pktgen-3.0.14.tar.gz

    tar xfz pktgen-3.0.14.tar.gz

    yum install libpcap libpcap-devel

    export RTE_SDK=<DPDK_DIR>

    export RTE_TARGET=x86_64-native-linuxapp-gcc

    make -j 8

    Tuning

    It is also recommended to go over Getting started with Performance Tuning of Mellanox adapters before starting.

    In addition, the following tuning parameters were set:

    • Disable flow control (pause frames) - we don't assume RoCE in this test.
    • Set PCIe MaxReadRequest to 1024
    • Set CPU frequency governer to performance mode

    A script (mlnx_dpdk_tune.sh) is attached as an example.

     

    Test Setup

    All the measurements are done with:

    • Host : Dell T620 (two sockets of E5-2697 v2 @ 2.70GHz)
    • CentOS Linux release 7.2.1511
    • ConnectX-4 100GbE dual port (FW12.14.2036) on NUMA node 0
    • pktgen-3.0.14
    • DPDK 16.07
    • MLNX_OFED_LINUX-3.3-1.0.4.0
    • Ethernet/IPv4/TCP varying IP addresses and ports (with command 'range all enable')
    • Disable Rx by mapping master core (lcore 0)

     

    Example

    The following command is an example to run pktgen-dpdk. To get the best performance, NUMA topology should be considered in coremask and txq_inline should be enabled.

    sudo ./app/app/x86_64-native-linuxapp-gcc/pktgen -c 0xfffff -w 3:00.0,txq_inline=256,txqs_min_inline=4 -w 3:00.1,txq_inline=256,txqs_min_inline=4 -- -T -P -m "[0:1-8].[0-1]"

     

    [0:1-8].[0-1] to allocate 8 Tx cores for two ports. port_map_string is explained here:

    Pktgen Commandline Options — Pktgen 2.7.7 documentation

     

    The below is an output example:

    Ports 0-1 of 2:  <Main Page>  Copyright (c) <2010-2016>, Intel Corporation10432

    Flags:Port    : P--------------:0 P--------------:1 0/24148

    Link State :      <UP-100000-FD>      <UP-100000-FD>       ----TotalRate----

    Pkts/s Max/Rx :                 0/0                 0/0                     0/0

           Max/Tx   : 32888608/32735228 0/0       32869092/32735228

    MBits/s Rx/Tx :             0/24093                 0/0                 0/24093

    Broadcast :                   0                   0

    Multicast :                   0                   0

      64 Bytes      :             0                   0

      65-127        :             0                   0

      128-255       :             0                   0

      256-511       :             0                   0

      512-1023      :             0                   0

      1024-1518     :             0                   0

    Runts/Jumbos :               0/0                 0/0

    Errors Rx/Tx :               0/0                 0/0

    Total Rx Pkts :               0                   0

          Tx Pkts   :         221812353               0

          Rx MBs    :             0                   0

          Tx MBs    :        163253                   0

    ARP/ICMP Pkts :              0/0                 0/0

    Src/Dest Port :         1234 / 5678         1234 / 5678

    Pattern Type :             abcd...             abcd...

    Tx Count/% Rate : Forever / 100%      Forever / 100%

    PktSize/Tx Burst:           76 /   32 76 /   32

    Src/Dest Port :         1234 / 5678         1234 / 5678

    Pkt Type:VLAN ID: IPv4 / TCP:0001     IPv4 / TCP:0001

    Dst  IP Address :         192.168.1.1         192.168.0.1 ---------------------

    Src  IP Address :      192.168.0.1/24      192.168.1.1/24

    Dst MAC Address : 7c:fe:90:57:a7:05 7c:fe:90:57:a7:04

    Src MAC Address 7c:fe:90:57:a7:04 7c:fe:90:57:a7:05

      -- Pktgen Ver: 3.0.02 (RTE 2.2.0) Powered by Intel® DPDK ---------------------

     

    How to Measure Ingress Rate

    The pktgen-dpdk is also designed to receive packets and count it to show rates. It classifies ingress traffic and shows statistics according to packet size. However, packet reception will consume a significant portion of the host performance and can impact transmission performance. Therefore, it is recommended to use an independent script such as the attached (eth_stat.sh), which is an example script for getting statistics from devices (ethtool -S) and calculating packets-per-second and bits-per-second.

     

    For example,

    # ./eth_stat.sh

    [enp3s0f1] / Rx:   49,936,356 pps    33,557,229,549 bps / Tx:            0 pps                 0 bps

    [enp3s0f0] / Rx:   49,904,571 pps    33,535,872,779 bps / Tx:            0 pps                 0 bps

    [enp3s0f1] / Rx:   49,985,777 pps    33,590,442,994 bps / Tx:            0 pps                 0 bps

    [enp3s0f0] / Rx:   50,008,097 pps    33,605,441,257 bps / Tx:            0 pps                 0 bps

    [enp3s0f1] / Rx:   50,018,018 pps    33,612,108,108 bps / Tx:            0 pps                 0 bps

    [enp3s0f0] / Rx:   49,979,688 pps    33,586,350,464 bps / Tx:            0 pps                 0 bps

    [enp3s0f1] / Rx:   50,040,416 pps    33,627,158,900 bps / Tx:            0 pps                 0 bps

     

    This shows Rx and Tx rate per second. As it is a bash script, it can not be very accurate but it is quite useful to roughly measure packet rates or for traffic monitor/log.

     

    Summary

    The packet generator (pktgen-dpdk) is capable of generating 100Mpps consisting of 64B packets and 100Gbps large packets with ConnectX-4 100GbE card while creating multiple flows by varying 5 tuple. This could be a good alternative to expensive commercial packet testers.