Getting started with ConnectX-4 100Gb/s Adapter for Linux

Version 7

    This post provides basic steps on how to configure and setup basic parameters for the Mellanox ConnectX-4 100Gb/s Adapter.

    This post is basic and is meant for beginners.

     

    References

    • MLNX_OFED User Manual

     

    Setup

    The basic setup consist of:

    • Two servers equipped with PCI gen3x16 slots
    • Two Mellanox ConnectX-4 adapter cards
    • One 100Gb/s Cable

     

    In this specific setup, CentOS 7.1 was installed on the servers.

     

    Prerequisites

    In case you plan to run performance test, it is recommended to tune the BIOS to high performance.

    Refer to Mellanox Tuning Guide and see this example: BIOS Performance Tuning Example.

     

    Configuration

    1. Install the latest MLNX_OFED (3.0.1 and above)

     

    2. Check that the adapter is recognized. Run the lspci command:

    # lspci | grep Mellanox

    05:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]

    05:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]

     

    Note: In ConnectX-4, each port is represented in a different but number.

     

    3. Change the link protocol to Ethernet using the MST mlxconfig tool.

        Note: The default link protocol for ConnectX-4 is InfiniBand.

     

        a. Start MST.

    # mst start

    Starting MST (Mellanox Software Tools) driver set

    Loading MST PCI module - Success

    Loading MST PCI configuration module - Success

    Create devices

    Unloading MST PCI module (unused) - Success

     

         b. Extract the vendor_part_id parameter.

             Note: ConnectX-4's ID is 4115

    # ibv_devinfo  | grep vendor_part_id

      vendor_part_id: 4115

      vendor_part_id: 4115

     

         c. Query the Host about ConnectX-4 adapters:

    # mlxconfig -d /dev/mst/mt4115_pciconf0 q

     

    Device #1:

    ----------

     

    Device type:    ConnectX4      

    PCI device:     /dev/mst/mt4115_pciconf0

     

     

    Configurations:          Current

             LINK_TYPE_P1    1      

             LINK_TYPE_P2    1

     

          Note that the LINK_TYPE_P1 and LINK_TYPE_P2 equal 1 (InfiniBand) by default.

     

         d. Change the port type to Ethernet (LINK_TYPE = 2):

    #mlxconfig -d /dev/mst/mt4115_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

     

    Device #1:

    ----------

     

    Device type:    ConnectX4      

    PCI device:     /dev/mst/mt4115_pciconf0

     

    Configurations:          Current New

             LINK_TYPE_P1    1       2      

             LINK_TYPE_P2    1       2      

     

    Apply new Configuration? ? (y/n) [n] : y

    Applying... Done!

    -I- Please reboot machine to load new configurations.

     

         e. Reboot the server.

     

     

    4. Configure IPs and MTU on both servers.

     

    Server S5:

    # ifconfig ens785f0  12.12.12.5/24 up
    # ifconfig ens785f0  mtu 9000

     

    Server S6:

    # ifconfig ens785f0 12.12.12.6/24 up
    # ifconfig ens785f0  mtu 9000

     

    5. After reboot, check that the port type was changed to Ethernet:

    # ibdev2netdev

    mlx5_0 port 1 ==> ens785f0 (Up)

    mlx5_1 port 1 ==> ens785f1 (Up)

     

    6. Make sure to disable firewall, iptables, SELINUX and other security proccesses that might block the traffic.

    # service firewalld stop

    # systemctl disable firewalld

    # service iptables stop

     

    Disable SELINUX on this file: /etc/selinux/config

     

    7. Run basic iperf test over:

     

    The following output is using the automation iperf script given in  HowTo Install iperf and Test Mellanox Adapters Performance.

    # ./iperf.sh

    ------------------------------------------------------------

    Server listening on TCP port 5001

    TCP window size: 85.3 KByte (default)

    ------------------------------------------------------------

    ------------------------------------------------------------

    Client connecting to 12.12.12.5, TCP port 5001

    TCP window size: 95.8 KByte (default)

    ------------------------------------------------------------

    [  4] local 12.12.12.5 port 5001 connected with 12.12.12.6 port 42257

    [ 10] local 12.12.12.6 port 42264 connected with 12.12.12.5 port 5001

    [  4] local 12.12.12.6 port 42257 connected with 12.12.12.5 port 5001

    [  3] local 12.12.12.6 port 42258 connected with 12.12.12.5 port 5001

    [  5] local 12.12.12.6 port 42259 connected with 12.12.12.5 port 5001

    [  6] local 12.12.12.6 port 42260 connected with 12.12.12.5 port 5001

    [  7] local 12.12.12.6 port 42261 connected with 12.12.12.5 port 5001

    [  8] local 12.12.12.6 port 42262 connected with 12.12.12.5 port 5001

    [  9] local 12.12.12.6 port 42263 connected with 12.12.12.5 port 5001

    [  5] local 12.12.12.5 port 5001 connected with 12.12.12.6 port 42258

    [  6] local 12.12.12.5 port 5001 connected with 12.12.12.6 port 42259

    [  7] local 12.12.12.5 port 5001 connected with 12.12.12.6 port 42260

    [  8] local 12.12.12.5 port 5001 connected with 12.12.12.6 port 42261

    [  9] local 12.12.12.5 port 5001 connected with 12.12.12.6 port 42262

    [ 10] local 12.12.12.5 port 5001 connected with 12.12.12.6 port 42263

    [ 11] local 12.12.12.5 port 5001 connected with 12.12.12.6 port 42264

    [ ID] Interval       Transfer     Bandwidth

    [ 10]  0.0-25.0 sec  36.9 GBytes  12.7 Gbits/sec

    [  4]  0.0-25.0 sec  36.9 GBytes  12.7 Gbits/sec

    [  3]  0.0-25.0 sec  37.3 GBytes  12.8 Gbits/sec

    [  5]  0.0-25.0 sec  34.5 GBytes  11.9 Gbits/sec

    [  6]  0.0-25.0 sec  33.3 GBytes  11.4 Gbits/sec

    [  7]  0.0-25.0 sec  34.2 GBytes  11.7 Gbits/sec

    [  8]  0.0-25.0 sec  44.2 GBytes  15.2 Gbits/sec

    [  9]  0.0-25.0 sec  30.4 GBytes  10.5 Gbits/sec

    [SUM]  0.0-25.0 sec   288 GBytes  98.9 Gbits/sec

    [ ID] Interval       Transfer     Bandwidth

    [  4]  0.0-25.0 sec  36.9 GBytes  12.7 Gbits/sec

    [  5]  0.0-25.0 sec  37.3 GBytes  12.8 Gbits/sec

    [  6]  0.0-25.0 sec  34.5 GBytes  11.9 Gbits/sec

    [  7]  0.0-25.0 sec  33.3 GBytes  11.4 Gbits/sec

    [  8]  0.0-25.0 sec  34.2 GBytes  11.7 Gbits/sec

    [  9]  0.0-25.0 sec  44.2 GBytes  15.2 Gbits/sec

    [ 10]  0.0-25.0 sec  30.4 GBytes  10.5 Gbits/sec

    [ 11]  0.0-25.0 sec  36.9 GBytes  12.7 Gbits/sec

    [SUM]  0.0-25.0 sec   288 GBytes  98.9 Gbits/sec