Using ConnectX-3 Pro with VMA over Ubuntu 16.04 Inbox Driver

Version 14

    This post helps you get VMA installed and running on a clean Ubuntu 16.04 using the Ubuntu inbox drivers.

     

     

    References

     

    Configuration and Setup

    Login as ROOT and set the system as follows while installing Ubuntu 16.04

     

    1. Make sure the kernel version is equal of greater than 4.4.

    $ uname -r

    4.4.0-2-generic

    $ cat /etc/lsb-release

    DISTRIB_ID=Ubuntu

    DISTRIB_RELEASE=16.04

    DISTRIB_CODENAME=xenial

    DISTRIB_DESCRIPTION="Ubuntu Xenial Xerus (development branch)"

     

    2. Make sure ConnectX-3 Pro is connected properly.

    $  lspci | grep Mellanox

    02:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

     

    Install Mellanox Drivers and Tools

    1. Install Basic drivers for user space access for Ethernet, RDMA, and Infiniband as follows:

    $ apt-get install dkms infiniband-diags libibverbs* ibacm librdmacm* libmlx4* libmlx5* mstflint libibcm.* libibmad.* libibumad* opensm srptools libmlx4-dev librdmacm-dev rdmacm-utils ibverbs-utils perftest vlan ibutils

     

    2.   Install Additional Build Tools.

    $ apt-get install libtool autoconf automake build-essential ibutils ibverbs-utils rdmacm-utils infiniband-diags perftest librdmacm-dev libibverbs-dev libmlx4-1 numactl libnuma-dev autoconf automake gcc g++ git libtool pkg-config

     

    3. Install libnl library.

    $ apt-get install libnl-3-200 libnl-route-3-200 libnl-route-3-dev libnl-utils

     

    4. Add modules to the kernel as follows:

    $ echo mlx4_ib >> /etc/modules

    $ echo ib_umad >> /etc/modules

    $ echo ib_cm >> /etc/modules

    $ echo ib_ucm >> /etc/modules

    $ echo rdma_ucm >> /etc/modules

     

    5. Load InfiniBand-related drivers.

    $ modprobe mlx4_ib; modprobe ib_umad; modprobe ib_cm; modprobe ib_ucm; modprobe rdma_ucm

     

    6. Install MST tools - click  here.

     

    7. Start MST:

    $ mst start

    Starting MST (Mellanox Software Tools) driver set

    Loading MST PCI module - Success

    Loading MST PCI configuration module - Success

    Create devices

    Unloading MST PCI module (unused) - Success

     

    8. Get PCI details:

    $ mst status

    MST modules:

    ------------

        MST PCI module loaded

        MST PCI configuration module loaded

     

     

    MST devices:

    ------------

    /dev/mst/mt4103_pciconf0         - PCI configuration cycles access.

                                       domain:bus:dev.fn=0000:02:00.0 addr.reg=88 data.reg=92

                                       Chip revision is: 00

    /dev/mst/mt4103_pci_cr0          - PCI direct access.

                                       domain:bus:dev.fn=0000:02:00.0 bar=0xc7200000 size=0x100000

                                       Chip revision is: 00

     

    9. Change the porttype to Ethernet.

    $ mlxconfig -d /dev/mst/mt4103_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

    $ reboot

     

    10. Verify drivers and ConnectX-3 are up.

    $ ibstat

    CA 'mlx4_0'

            CA type: MT4103

            Number of ports: 2

            Firmware version: 2.34.5000

            Hardware version: 0

            Node GUID: 0xe41d2d03001d6690

            System image GUID: 0xe41d2d03001d6693

            Port 1:

                    State: Down

                    Physical state: Polling

                    Rate: 10

                    Base lid: 0

                    LMC: 0

                    SM lid: 0

                    Capability mask: 0x02504868

                    Port GUID: 0xe41d2d03001d6691

                    Link layer: InfiniBand

            Port 2:

                    State: Down

                    Physical state: Polling

                    Rate: 10

                    Base lid: 0

                    LMC: 0

                    SM lid: 0

                    Capability mask: 0x02504868

                    Port GUID: 0xe41d2d03001d6692

                    Link layer: InfiniBand

     

    $ ibv_devinfo

    hca_id: mlx4_0

            transport:                      InfiniBand (0)

            fw_ver:                         2.34.5000

            node_guid:                      e41d:2d03:001d:6690

            sys_image_guid:                 e41d:2d03:001d:6693

            vendor_id:                      0x02c9

            vendor_part_id:                 4103

            hw_ver:                         0x0

            board_id:                       MT_1090111019

            phys_port_cnt:                  2

                    port:   1

                            state:                  PORT_DOWN (1)

                            max_mtu:                4096 (5)

                            active_mtu:             4096 (5)

                            sm_lid:                 0

                            port_lid:               0

                            port_lmc:               0x00

                            link_layer:             InfiniBand

     

                    port:   2

                            state:                  PORT_DOWN (1)

                            max_mtu:                4096 (5)

                            active_mtu:             4096 (5)

                            sm_lid:                 0

                            port_lid:               0

                            port_lmc:               0x00

                            link_layer:             InfiniBand

     

    Install VMA

    1. Download libvma source code.

    $ git clone https://github.com/Mellanox/libvma.git $ cd libvma

     

    2. Build and Install libvma

    $ ./autogen.sh
    $ ./configure --prefix=/usr
    $ make -j
    $ make install
    $ ldconfig

     

    3. Enable Flow Steering

    $ echo options mlx4_core log_num_mgm_entry_size=-1 > /etc/modprobe.d/mlnx.conf

     

    4. Restart the Driver

    $ modprobe -r mlx4_en

    $ modprobe -r mlx4_ib

    $ modprobe -r mlx4_core

    $ modprobe mlx4_core

    $ modprobe mlx4_ib

    $ modprobe mlx4_en

     

    5. Enable New Limits for the User

    $ ulimit -l unlimited

     

    6. Provide Memory Settings for High Performance and RDMA. Configure huge-pages for better libvma performance:

    $ echo 1000000000 > /proc/sys/kernel/shmmax
    $ echo 800 > /proc/sys/vm/nr_hugepage

     

    7. Load libvma and Run the Application (as root)

    $ LD_PRELOAD=libvma.so <test_app>