This document describes how to install and test Mellanox OFED for Linux package on a single physical/virtual machine with Mellanox ConnectX®-5 adapter card installed.
For more information click on Mellanox OFED for Linux User Manual.
Downloading Mellanox OFED
- Verify that the system has a Mellanox network adapter (HCA/NIC) installed.
# lspci -v | grep MellanoxThe following example shows a system with an installed Mellanox HCA:
- Download the ISO image according to you OS to your servers share folder.
The image’s name has the format
MLNX_OFED_LINUX-<ver>-<OS label><CPUarch>.iso. You can download it from:
http://www.mellanox.com > Products > Software > InfiniBand/VPI Drivers > Mellanox OFED Linux (MLNX_OFED) > Download.
- Use the MD5SUM utility to confirm the downloaded file’s integrity. Run the following command and compare the result to the value provided on the download page.
# md5sum MLNX_OFED_LINUX-<ver>-<OS label>.iso
Installing Mellanox OFED
MLNX_OFED is installed by running themlnxofedinstall script. The installation script, performs the following:
- Discovers the currently installed kernel
- Uninstalls any software stacks that are part of the standard operating system distribution or another vendor's commercial stack
- Installs the MLNX_OFED_LINUX binary RPMs (if they are available for the current kernel)
- Identifies the currently installed InfiniBand and Ethernet network adapters and automatically upgrades the firmware
The installation script removes all previously installed Mellanox OFED packages and re-installs from scratch. You will be prompted to acknowledge the deletion of the old packages.
- Log into the installation machine as root.
- Copy the downloaded ISO to /root
- Mount the ISO image on your machine.
# mkdir /mnt/iso# mount -o loop /share/MLNX_OFED_LINUX-4.2-188.8.131.52-ubuntu16.04-x86_64.iso /mnt/iso# cd /mnt/iso
- Run the installation script.
- Reboot after the installation finished successfully.
# /etc/init.d/openibd restart# rebootBy default both ConnectX®-5 VPI ports are initialized as InfiniBand ports.
ConnectX®-5 ports can be individually configured to work as InfiniBand or Ethernet ports.
- Check the ports’ mode is Ethernet
- If you see the following, change the interfaces port type to Ethernet
Change the interfaces port type to Ethernet mode.
Use the mlxconfig script after the driver is loaded.
* LINK_TYPE_P1=2 is a Ethernet modea. Start mst and see ports names
# mst start
# mst statusb. Change the mode of 1 port to Ethernet:
# mlxconfig -d /dev/mst/mt4121_pciconf0 s LINK_TYPE_P1=2c. Query the Ethernet devices and print the information available to use it from the userspace.
Port 1 set to Ethernet mode
- Run the ibdev2netdev utility to see all the associations between the Ethernet devices and the IB devices/ports.
# ifconfig enp1f0 184.108.40.206 netmask 255.255.255.0
- Insert to the /etc/network/interfaces file the lines below after the following lines:
# vim /etc/network/interfacesThe new lines:
iface eno0 inet dhcp
iface enp1f0 inet static
# vim /etc/network/interfaces
iface eno0 inet dhcp
iface eenp1f0 inet static
- Check the network configuration is set correctly.
# ifconfig -a
(OPTIONAL) Lossless fabric with L3(DSCP) configuration
RDMA has been initially developed for InfiniBand networks which are inherently lossless. They incorporate a link level flow control to ensure that packets are not dropped within the fabric.
RoCE implements the RDMA protocol over a standard Ethernet/IP network, which can be lossy.
Due to the performance implications of a lossy network when running RoCE, it is recommended to enable flow control within your fabric.
- Check the flow control settings for Mellanox network adapters by running the command:
ethtool -aYou shall get RX and TX set off. If the RX and TX settings are turned on, as shown below, then they should be disabled:
<mlnx interface name>
# ethtool -A enp1f0 rx off tx off
# ethtool -a enp1f0
Pause parameters for enp1f0:
- Follow the procedure in this post: Lossless RoCE Configuration for Linux Drivers in DSCP-Based QoS Mode
It provides a configuration example for Mellanox devices installed with MLNX_OFED running RoCE over a lossless network, in DSCP-based QoS mode.