How to Install Mellanox OFED on Linux (Rev 4.4-

Version 3

    This document describes how to install and test Mellanox OFED for Linux package on a single physical/virtual machine with Mellanox ConnectX®-5 adapter card installed.

    For more information click on Mellanox OFED for Linux User Manual.



    Downloading Mellanox OFED

    1. Verify that the system has a Mellanox network adapter (HCA/NIC) installed.
      # lspci -v | grep Mellanox
      The following example shows a system with an installed Mellanox HCA:
    2. Download the ISO image according to you OS to your servers share folder.
      The image’s name has the format
      MLNX_OFED_LINUX-<ver>-<OS label><CPUarch>.iso. You can download it from: > Products > Software > InfiniBand/VPI Drivers > Mellanox OFED Linux (MLNX_OFED) > Download.
    3. Use the MD5SUM utility to confirm the downloaded file’s integrity. Run the following command and compare the result to the value provided on the download page.
      # md5sum MLNX_OFED_LINUX-<ver>-<OS label>.iso

    Installing Mellanox OFED

    MLNX_OFED is installed by running themlnxofedinstall script. The installation script, performs the following:

    • Discovers the currently installed kernel
    • Uninstalls any software stacks that are part of the standard operating system distribution or another vendor's commercial stack
    • Installs the MLNX_OFED_LINUX binary RPMs (if they are available for the current kernel)
    • Identifies the currently installed InfiniBand and Ethernet network adapters and automatically upgrades the firmware

    The installation script removes all previously installed Mellanox OFED packages and re-installs from scratch. You will be prompted to acknowledge the deletion of the old packages.

    1. Log into the installation machine as root.
    2. Copy the downloaded ISO to /root
    3. Mount the ISO image on your machine.
      # mkdir /mnt/iso# mount -o loop /share/MLNX_OFED_LINUX-4.2- /mnt/iso# cd /mnt/iso
    4. Run the installation script.
      # ./mlnxofedinstall
    5. Reboot after the installation finished successfully.
      # /etc/init.d/openibd restart# reboot
      By default both ConnectX®-5 VPI ports are initialized as InfiniBand ports.
      ConnectX®-5 ports can be individually configured to work as InfiniBand or Ethernet ports.
    6. Check the ports’ mode is Ethernet
      # ibv_devinfo
    7. If you see the following, change the interfaces port type to Ethernet

      Change the interfaces port type to Ethernet mode.
      Use the mlxconfig script after the driver is loaded.
      * LINK_TYPE_P1=2 is a Ethernet modea. Start mst and see ports names
      # mst start
      # mst status
      b. Change the mode of 1 port to Ethernet:
      # mlxconfig -d /dev/mst/mt4121_pciconf0 s LINK_TYPE_P1=2
      Port 1 set to Ethernet mode
      # reboot
      c. Query the Ethernet devices and print the information available to use it from the userspace.
      # ibv_devinfo
    8. Run the ibdev2netdev utility to see all the associations between the Ethernet devices and the IB devices/ports.
      # ibdev2netdev
      # ifconfig enp1f0 netmask
    9. Insert to the /etc/network/interfaces file the lines below after the following lines:
      # vim /etc/network/interfaces

      auto eno0
      iface eno0 inet dhcp
      The new lines:
      auto enp1f0
      iface enp1f0 inet static
      address 31.31.31.xx
      # vim /etc/network/interfaces
      auto eno0

      iface eno0 inet dhcp
      auto enp1f0
      iface eenp1f0 inet static
    10. Check the network configuration is set correctly.
      # ifconfig -a

    (OPTIONAL) Lossless fabric with L3(DSCP) configuration

    RDMA has been initially developed for InfiniBand networks which are inherently lossless. They incorporate a link level flow control to ensure that packets are not dropped within the fabric.
    RoCE implements the RDMA protocol over a standard Ethernet/IP network, which can be lossy.
    Due to the performance implications of a lossy network when running RoCE, it is recommended to enable flow control within your fabric.

    1. Check the flow control settings for Mellanox network adapters by running the command: ethtool -a <mlnx interface name>
      You shall get RX and TX set off. If the RX and TX settings are turned on, as shown below, then they should be disabled:
      # ethtool -A enp1f0 rx off tx off
      # ethtool -a enp1f0
      Pause parameters for enp1f0:
      Autonegotiate: off
      RX: off
      TX: off
    2. Follow the procedure in this post: Lossless RoCE Configuration for Linux Drivers in DSCP-Based QoS Mode
      It provides a configuration example for Mellanox devices installed with MLNX_OFED running RoCE over a
      lossless network, in DSCP-based QoS mode.