HowTo Configure VMA Inside Docker Container (RHEL7.2 inbox driver)

Version 7

    This post describes how to create and run docker containers with Mellanox's Messaging Accelerator (VMA), using RHEL7.2, Docker 1.9 and ConnectX-3 with inbox driver.

     

     

    References

     

    Host Installation and Configuration

    1. Install the required packages.

    # yum groupinstall -y "Infiniband Support"

    # yum install libibverbs-devel librdmacm-devel libmlx4

    # yum --setopt=group_package_types=optional install libnl3-devel

        Install libnl3-devel. This package is part of the rhel-7-server-optional-rpms repository, which require Red Hat subscription (https://access.redhat.com/documentation/en-US/Red_Hat_Subscription_Management/1/html/RHSM/supplementary-repos.html). The package is also available online in several sites.

     

    2. Enable Flow Steering on the device. Add the following line to /etc/modprobe.d/mlnx.conf.

    options mlx4_core log_num_mgm_entry_size=-1

     

    3. Reload the modules.

    # modprobe -rv mlx4_en mlx4_ib mlx4_core

    # modprobe -v mlx4_en mlx4_ib mlx4_core rdma_ucm

     

    4. Configure hugepages for better VMA performance (recommended).

    # echo 800 > /proc/sys/vm/nr_hugepages

        By default, memory consumption is ~800MB per VMA process. The above command allocates 800 hugepages and assume the default of 2MB per page. If larger pages are used (1GB pages for example), lower the number of pages needed.

        Use /proc/meminfo | grep Huge to see the hugepages information of the system.

     

    Build VMA on the Host

    Note: Building VMA inside the container requires large amount of packages which are needed for this purpose alone. Therefore, in order to keep the container image minimal, building the VMA library can be done on the host.

    # git clone https://github.com/Mellanox/libvma.git

    # cd libvma

    # ./autogen.sh

    # ./configure --prefix=/usr

    # make –j

    The libvma.so.7.0.11 library should be created under /src/vma/.libs/

     

    Important note: Same version of the user space libraries (libibverbs, librdmacm, libmlx4 and libnl3) should be used inside the container and on the host.

     

    Docker Installation

    1. Create the Docker repository. Add the /etc/yum.repos.d/docker.repo file with the following content.

    [dockerrepo]

    name=Docker Repository

    baseurl=https://yum.dockerproject.org/repo/main/centos/7

    enabled=1

    gpgcheck=1

    gpgkey=https://yum.dockerproject.org/gpg

    See: Docker Installation on Red Hat Enterprise Linux from docker website

     

    2. Install Docker, start and enable the service to run at startup.

    # yum install docker-engine

    # systemctl start docker

    # systemctl enable docker

     

    3. Verify installation was completed successfully.

    # docker run hello-world

     

    Create a Docker Container Image (with VMA)

    1. Download the Docker RHEL7 image and verify.

    # docker pull rhel7

    # docker images

     

    2. Create a directory to be mounted inside the container and move libvma and the configuration file (libvma/src/vma/util/libvma.conf) inside the directory.

    # mkdir /tmp/mnt

    # cp /var/soft/libvma/src/vma/.libs/libvma.so.7.0.11 /tmp/mnt

    # cp /var/soft/libvma/src/vma/util/libvma.conf /tmp/mnt

     

    3. Run the container, mount the repository directories (if needed) and the directory with libvma from the host inside the container.

    # docker run -it -v /tmp/mnt:/tmp/mnt -v /etc/yum.repos.d:/etc/yum.repos.d -v /etc/pki/rpm-gpg:/etc/pki/rpm-gpg rhel7 /bin/bash

         The -v <host dir>:<container dir> option mounts the host <host dir> to <container dir> inside the container.

     

    4. Install the required packages for VMA (inside the container).

    (container)# yum install libmlx4 libibverbs librdmacm libnl3 libibverbs-devel librdmacm-devel

     

    5. Copy the libvma library to /usr/lib (inside the container) and create symbolic links.

    (container)# cp /tmp/mnt/libvma.so.7.0.11 /usr/lib/

    (container)# cd /usr/lib

    (container)# ln -s libvma.so.7.0.11 libvma.so

    (container)# ln -s libvma.so.7.0.11 libvma.so.7

    (container)# ldconfig

     

    6. Copy the libvma configuration file to /etc/libvma.conf.

        By default, all the lines in the file are commented out. This is a good practice for future use and to avoid unnecessary VMA warnings.

    (container)# cp /tmp/mnt/libvma.conf /etc/

     

    7. (Optional) Install the sockperf package (inside the container).

    (container)# yum install sockperf

     

    8. Save the container's image.

       The following steps are used to backup the container's image.

     

       a. Exit the container (using exit command from the container).

    (container)# exit

     

      b. Get the container ID (on the host) and save the new image with the new name.

    #docker ps -l

    CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                        PORTS               NAMES

    9191562cba07        rhel7               "/bin/bash"         12 minutes ago      Exited (130) 31 seconds ago                       happy_mirzakhani

    #docker commit -m="Added VMA Eth" -a="user1" 9191562cba07 rhel7-vma

     

      c. Verify the new image was saved.

    # docker images

    REPOSITORY          TAG                 IMAGE ID            CREATED              VIRTUAL SIZE

    rhel7-vma           latest              77d13a9708e7        About a minute ago   277.7 MB

    rhel7               latest              6883d5422f4e        4 weeks ago          201.7 MB

    hello-world         latest              0a6ba66e537a        8 weeks ago          960 B

     

    ulimit Consideration

    See HowTo Configure VMA inside Docker Container (with MLNX_OFED) - ulimit Consideration

     

    Verification

    The steps below are to ensure that VMA is working within the container.

    1. Run sockperf server on a different server (We used VMA also here but this is not a must).

    # LD_PRELOAD=libvma.so sockperf server

     

    2. Run the Docker container.

    # docker run -t -i --net=host --ulimit memlock=-1 --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm rhel7-vma /bin/bash

       This command allows the container to access the InfiniBand devices uverbs0 and the rdma-cm required by VMA. --ulimit is needed if not set as a daemon option.

     

    3. Run the sockperf client with the VMA library (inside the container).

    (container)# LD_PRELOAD=libvma.so sockperf ping-pong -i 192.168.70.8 -t 5

    VMA INFO   : ---------------------------------------------------------------------------

    VMA INFO   : VMA_VERSION: 7.0.11-0 Development Snapshot built on Dec  8 2015 15:29:22

    VMA INFO   : Cmd Line: sockperf ping-pong -i 192.168.70.8 -t 5

    VMA INFO   : Current Time: Wed Dec  9 10:57:02 2015

    VMA INFO   : Pid:    51

    VMA INFO   : OFED Version:  VMA INFO   : Architecture: x86_64

    VMA INFO   : Node: r-bd09.mtr.labs.mlnx

    VMA INFO   : ---------------------------------------------------------------------------

    VMA INFO   : Log Level                      3                          [VMA_TRACELEVEL]

    VMA INFO   : CQ Interrupts Moderation       Disabled                   [VMA_CQ_MODERATION_ENABLE]

    VMA INFO   : CQ Adaptive Moderation         Disabled                   [VMA_CQ_AIM_INTERVAL_MSEC]

    VMA INFO   : ---------------------------------------------------------------------------

    sockperf: == version #2.6 ==

    sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)

     

    [ 0] IP = 192.168.70.8    PORT = 11111 # UDP

    sockperf: Warmup stage (sending a few dummy messages)...

    sockperf: Starting test...

    sockperf: Test end (interrupted by timer)

    sockperf: Test ended

    sockperf: [Total Run] RunTime=5.100 sec; SentMessages=2072936; ReceivedMessages=2072935

    sockperf: ========= Printing statistics for Server No: 0

    sockperf: [Valid Duration] RunTime=5.000 sec; SentMessages=2036150; ReceivedMessages=2036150

    sockperf: ====> avg-lat=  1.212 (std-dev=0.272)

    sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0

    sockperf: Summary: Latency is 1.212 usec

    sockperf: Total 2036150 observations; each percentile contains 20361.50 observations

    sockperf: ---> <MAX> observation =   38.826

    sockperf: ---> percentile  99.99 =    6.858

    sockperf: ---> percentile  99.90 =    2.961

    sockperf: ---> percentile  99.50 =    2.338

    sockperf: ---> percentile  99.00 =    2.023

    sockperf: ---> percentile  95.00 =    1.776

    sockperf: ---> percentile  90.00 =    1.361

    sockperf: ---> percentile  75.00 =    1.183

    sockperf: ---> percentile  50.00 =    1.141

    sockperf: ---> percentile  25.00 =    1.113

    sockperf: ---> <MIN> observation =    1.057

    Note: This is not an optimized VMA run, just a sanity check. VMA has many optimization options but they are outside the scope of this post.

    Note: The results above are similar to sockperf results over VMA when running the same command outside the container.