Getting Started with ConnectX-5 100Gb/s Adapters for Linux

Version 4

    This post provides basic steps on how to configure and set up basic parameters for the Mellanox ConnectX-5 100Gb/s adapter.

    This post is basic and is meant for beginners. The procedure is very similar to the one for the ConnectX-4 adapter (in fact, it uses the same mlx5 driver).

     

    Note: ConnectX-5 adapters can be used only with MLNX_OFED rel. 4.0 or later installed.

     

    References

     

    Setup

    The basic setup consists of:

    • Two servers equipped with PCI gen3x16 slots
    • Two Mellanox ConnectX-5 adapter cards
    • One 100Gb/s cable

     

    In this specific setup, CentOS 7.2 was installed on the servers.

     

    Prerequisites

    If you plan to run performance tests, we recommend that you tune the BIOS to high performance.

    Refer to Mellanox Tuning Guide and see this example: BIOS Performance Tuning Example.

     

    Configuration

    1. Install the latest MLNX_OFED (rel. 4.0 and later).

     

    2. Check that the adapters are "recognized" by running the lspci command:

    # lspci | grep Mellanox82:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]82:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]

     

    Note: In ConnectX-5, each port is identified by a unique number.

     

    3. Change the link protocol to Ethernet using the MFT mlxconfig tool.

        Note: The default link protocol for ConnectX-4 is InfiniBand.

     

    a. Start MFT.

    # mst start

    Starting MST (Mellanox Software Tools) driver set

    Loading MST PCI module - Success

    Loading MST PCI configuration module - Success

    Create devices

    Unloading MST PCI module (unused) - Success

     

    b. Extract the vendor_part_id parameter. Note: ConnectX-5's ID is 4119.

    # ibv_devinfo  | grep vendor_part_id

      vendor_part_id: 4119

      vendor_part_id: 4119

     

    c. Query the Host about ConnectX-4 adapters:

    # mlxconfig -d /dev/mst/mt4119_pciconf0 q

     

    Device #1:

    ----------

     

    Device type:    ConnectX5      

    PCI device:     /dev/mst/mt4119_pciconf0

     

     

    Configurations:          Current

    ...

             LINK_TYPE_P1    1      

             LINK_TYPE_P2    1

    ....

     

          Note that the LINK_TYPE_P1 and LINK_TYPE_P2 equal 1 (InfiniBand) by default.

     

    d. Change the port type to Ethernet (LINK_TYPE = 2):

    #mlxconfig -d /dev/mst/mt4119_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

     

    Device #1:

    ----------

     

    Device type:    ConnectX5     

    PCI device:     /dev/mst/mt4119_pciconf0

     

    Configurations:          Current New

             LINK_TYPE_P1    1       2      

             LINK_TYPE_P2    1       2      

     

    Apply new Configuration? ? (y/n) [n] : y

    Applying... Done!

    -I- Please reboot machine to load new configurations.

     

    e. Reboot the server.

     

    4. Configure IPs and MTUs on both servers.

     

    For Server S5:

    # ifconfig ens801f0  15.15.15.5/24 up
    # ifconfig ens801f0  mtu 9000

     

    For Server S6:

    # ifconfig ens801f0  15.15.15.6/24 up
    # ifconfig ens801f0  mtu 9000

     

    5. After you reboot, check that the port type was changed to Ethernet for each:

    # ibdev2netdev

    mlx5_0 port 1 ==> ens801f0  (Up)

    mlx5_1 port 1 ==> ens801f0  (Up)

     

    6. Make sure that you disable the firewall, iptables, SELINUX, and other security processes that might block the traffic.

    # service firewalld stop

    # systemctl disable firewalld

    # service iptables stop

     

    Disable SELINUX in the config file located at: /etc/selinux/config.

     

    7. Run the basic iperf test again.

     

    The following output is displayed using the automation iperf script described in  HowTo Install iperf and Test Mellanox Adapters Performance.

     

    Run the iperf client process on one host with the iperf server:

    # iperf -s -P8

     

    Run the iperf client process on the other host with the iperf client:

    # iperf -c 15.15.15.6 -P8

    ------------------------------------------------------------

    Client connecting to 15.15.15.6, TCP port 5001

    TCP window size:  325 KByte (default)

    ------------------------------------------------------------

    [ 10] local 15.15.15.5 port 57522 connected with 15.15.15.6 port 5001

    [  4] local 15.15.15.5 port 57508 connected with 15.15.15.6 port 5001

    [  3] local 15.15.15.5 port 57510 connected with 15.15.15.6 port 5001

    [  6] local 15.15.15.5 port 57512 connected with 15.15.15.6 port 5001

    [  7] local 15.15.15.5 port 57514 connected with 15.15.15.6 port 5001

    [  5] local 15.15.15.5 port 57516 connected with 15.15.15.6 port 5001

    [  8] local 15.15.15.5 port 57518 connected with 15.15.15.6 port 5001

    [  9] local 15.15.15.5 port 57520 connected with 15.15.15.6 port 5001

    [ ID] Interval       Transfer     Bandwidth

    [ 10]  0.0-10.0 sec  13.6 GBytes  11.7 Gbits/sec

    [  3]  0.0-10.0 sec  13.9 GBytes  12.0 Gbits/sec

    [  6]  0.0-10.0 sec  18.6 GBytes  16.0 Gbits/sec

    [  7]  0.0-10.0 sec  10.9 GBytes  9.38 Gbits/sec

    [  5]  0.0-10.0 sec  14.7 GBytes  12.6 Gbits/sec

    [  8]  0.0-10.0 sec  16.0 GBytes  13.7 Gbits/sec

    [  9]  0.0-10.0 sec  17.2 GBytes  14.8 Gbits/sec

    [  4]  0.0-10.0 sec  9.92 GBytes  8.52 Gbits/sec

    [SUM]  0.0-10.0 sec   115 GBytes  98.6 Gbits/sec

     

    Troubleshooting

    1. If MLNX_OFED rel. 4.0 or later is not used, the card will be identified as a ConnectX-4 adapter by default.

    # ofed_info -s

    MLNX_OFED_LINUX-3.4-2.0.0.0:

     

    # lspci | grep Mellanox

    81:00.0 Infiniband controller: Mellanox Technologies MT28800 Family [ConnectX-4]

    81:00.1 Infiniband controller: Mellanox Technologies MT28800 Family [ConnectX-4]

     

     

    To correct this, install MLNX_OFED rel. 4.0 or later.

    # ofed_info -s

    MLNX_OFED_LINUX-4.0-0.1.5.0:

     

    # lspci | grep Mel

    81:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]

    81:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]

     

    2. Make sure that you run the iperf process from the root "/" folder.