Mellanox OFED Driver Installation and Configuration for SR-IOV

Version 16

    Note: This document is outdated, refer to the following documents:

     

     

    ---

    This post shows how to enable SR-IOV on Mellanox ConnectX-3 adapters.

     

    References


    In order to have SR-IOV enabled on your system, follow the Mellanox OFED installation and SR-IOV installation procedures in Mellanox OFED User Manual (detailed explanations).

     

    OFED Installation

     

    Before you start, make sure SR-IOV is enabled on the BIOS (In most cases it is enabled by default).

     

    To install MLNX_OFED with SR-IOV enabled run the following command:

    #./mlnxofedinstall --enable-sriov


    --enable-sriov: Burn SR-IOV enabled firmware.

     

    Note:

    1. The "total-vfs" parameter (Maximum number of Virtual Functions in SR-IOV mode) was removed from the OFED installation in OFED 2.1  - The default is 16
    2. SR-IOV
      1. --enable_sriov has effect only for Mellanox branded cards. Most other OEM branded users currently need to modify the INI.
      2. Some OEMs have already SRIOV enabled in their INIs (HP, Dell) with 8 or 16 functions.
      3. Changing the INI for an OEM card will violate the card’s warranty hence is not possible for many users. It can be done via the ‘mlxconfig’ tool (MFT) instead.


    For full OFED Installation refer to Mellanox OFED User Manual.

     

    Configuration

    1. Verify you are running OFED version 2.0.3 or higher (for SR-IOV support)

    # ofed_info |head

    MLNX_OFED_LINUX-2.0.3-0.0.0 (OFED-2.0.3-0.0.0):

    ar_mgr:

    /mswg/release/osm_plugins/ar_mgr/ar_mgr-1.0-0.9.g8b7dbc6.tar.gz

    bupc:

    /mswg/release/mlnx_ofed_upc/bupc-2.2-364.src.rpm

    cc_mgr:

    mswg/release/osm_plugins/cc_mgr/cc_mgr-1.0-0.8.g8b7dbc6.tar.gz

    2. Verify that SR-IOV is installed properly

    # mst start

    # flint -d /dev/mst/mt4099_pciconf0 dc

     

    This command will print the adapter configuration, look for total_vfs and sriov_en parameters.

    [HCA]

    hca_header_device_id = 0x1003

    hca_header_subsystem_id = 0x0006

    dpdp_en = true

    eth_xfi_en = true

    mdio_en_port1 = 0

    num_pfs = 1

    total_vfs = 16 // Total number of VMs

    sriov_en = true // SR-IOV is enabled

     

    if total_vfs and sriov_en are not in the output, it means that SR-IOV is not enabled on the firmware. Need to burn firmware with SR-IOV enabled.

     

    Note: to change the total_vfs parameter you need to change the ini file and re-burn the Firmware.

     

    3. After the installation perform the following changes:

      Create (or open) '/etc/modprobe.d/mlx4_core.conf' file and add (or update) the following entry:

    options mlx4_core port_type_array=2,2 num_vfs=16 probe_vf=0

     

      The mlnx4_core parameters are as follows:

    • port_type_array=1,2 (1= InfiniBand, 2 = Ethernet ).
    • num_vfs=16  Total Virtual functions
    • probe_vf=0  In case of Ethernet (with one port). The number of Ethernet interfaces the hypervisor has. Those interfaces can be given to any VM configured as paravirt interface.

     

      When running OpenStack/OpenFlow solution,

      The following parameters should be added to the 'options' as parameters to the 'mlx4_core' driver:

    • enable_64b_cqe_eqe=0
    • log_num_mgm_entry_size=-1
    options mlx4_core port_type_array=2,2 num_vfs=16 probe_vf=0 enable_64b_cqe_eqe=0  log_num_mgm_entry_size=-1

     

    4. Reboot the server:

    #reboot

     

    5. Change /boot/grub/menu.lst file - Add intel_iommu=on option:

    # grub.conf generated by anaconda

    #

    # Note that you do not have to rerun grub after making changes to this file

    # NOTICE:  You have a /boot partition.  This means that

    #          all kernel and initrd paths are relative to /boot/, eg.

    #          root (hd0,0)

    #          kernel /vmlinuz-version ro root=/dev/sda2

    #          initrd /initrd-[generic-]version.img

    #boot=/dev/sda

    default=0

    timeout=5

    splashimage=(hd0,0)/grub/splash.xpm.gz

    hiddenmenu

    title Red Hat Enterprise Linux (2.6.32-358.el6.x86_64)

            root (hd0,0)

            kernel /vmlinuz-2.6.32-358.el6.x86_64 ro root=UUID=387dda24-be0f-4de4-9176-63f3ea9eff01 rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_NO_LVM crashkernel=auto rhgb quiet rd_NO_DM rhgb quiet intel_iommu=on

            initrd /initramfs-2.6.32-358.el6.x86_64.img

    Note: the intel_iommu=on parameter should be on the same line of the kernel - it is just a long line.

    6. Restart the server


    7.  Verify that SR-IOV is installed using lspci command

    # lspci  | grep -i Mel

    06:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

    06:00.1 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

    06:00.2 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

    06:00.3 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

    06:00.4 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

    06:00.5 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

    06:00.6 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

    06:00.7 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

    06:01.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX--3 Virtual Function]