HowTo Install iperf and Test Mellanox Adapters Performance

Version 7

    This post shows a simple procedure on how to install iperf and test performance on Mellanox adapters.

    This post is basic and is meant for beginners.

     

    References

    • MLNX_OFED User Manual

     

    Setup

    Two hosts connected back to back or via a switch.

     

    Configuration

    1. Make sure you have two servers with IP link connectivity between them (ping is running).

     

    2. Download and install the iperf package from the git location (or any other distribution)

    # cd /tmp

    # wget http://downloads.sourceforge.net/project/iperf/iperf-2.0.5.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fiperf%2F&ts=1441315914&use_mirror=skylineservers iperf-2.0.5.tar.gz

    ...

    # tar xzvf iperf-2.0.5.tar.gz

    ...

    # cd iperf-2.0.5

    # ./configure

    ....

    # make

    ...

    # make install

    ...

     

    3. Make sure to disable firewall, iptables, SELINUX and other security processes that might block the traffic.

    # service firewalld stop

    # systemctl disable firewalld

    # service iptables stop

     

    Disable SELINUX on this file: /etc/selinux/config

     

    4. Run the iperf server process on one of the hosts (recommended with 8 or 16 threads)

    #iperf -s -P8

     

    5. Run the iperf client process on the other host with the iperf server IP address.

     

    Here is an output example from ConnectX-4 100Gb/s

    # iperf -c 12.12.12.6 -P8

    ------------------------------------------------------------

    Client connecting to 12.12.12.6, TCP port 5001

    TCP window size: 22.5 KByte (default)

    ------------------------------------------------------------

    [ 10] local 12.12.12.5 port 56143 connected with 12.12.12.6 port 5001

    [  6] local 12.12.12.5 port 56139 connected with 12.12.12.6 port 5001

    [  5] local 12.12.12.5 port 56138 connected with 12.12.12.6 port 5001

    [  4] local 12.12.12.5 port 56137 connected with 12.12.12.6 port 5001

    [  3] local 12.12.12.5 port 56136 connected with 12.12.12.6 port 5001

    [  8] local 12.12.12.5 port 56140 connected with 12.12.12.6 port 5001

    [  7] local 12.12.12.5 port 56141 connected with 12.12.12.6 port 5001

    [  9] local 12.12.12.5 port 56142 connected with 12.12.12.6 port 5001

    [ ID] Interval       Transfer     Bandwidth

    [ 10]  0.0-10.0 sec  15.4 GBytes  13.3 Gbits/sec

    [  6]  0.0-10.0 sec  15.3 GBytes  13.1 Gbits/sec

    [  5]  0.0-10.0 sec  15.4 GBytes  13.2 Gbits/sec

    [  4]  0.0-10.0 sec  3.22 GBytes  2.77 Gbits/sec

    [  3]  0.0-10.0 sec  13.6 GBytes  11.7 Gbits/sec

    [  8]  0.0-10.0 sec  15.4 GBytes  13.2 Gbits/sec

    [  7]  0.0-10.0 sec  15.4 GBytes  13.2 Gbits/sec

    [  9]  0.0-10.0 sec  15.4 GBytes  13.2 Gbits/sec

    [SUM]  0.0-10.0 sec   109 GBytes  94.1 Gbits/sec

     

    Automation

    Here is a script example that you can copy&paste to a file on one of the servers.

     

    • PTH - number of iperf threads
    • TIME - time in seconds
    • remote-server - remote server name
    • 12.12.12.5 - local IP example, this is the IP on the local server on the Mellanox adapter.

     

    The script will run iperf server on the local machine, and connect via SSH to the remote machine and then run the iperf client to the local machine.

     

    Note: This script assumes the NUMA architecture, and the adapter are binded to numa 0.

     

    Note: Make sure that you can perform SSH login without entering your the password.

     

    PTH=8

    TIME=10

     

    numactl --cpunodebind=0 iperf -s -t $TIME -P $PTH & \

    ssh remote-server "numactl --cpunodebind=0 iperf -c 12.12.12.5 -t $TIME -P $PTH "