How-To Configure Mellanox VMA inside Docker Container using Docker Composer

Version 2

    This post describes how to install Docker container,  how to configure Mellanox VMA Kernel Bypass inside Docker, and how to create multiple Docker images with Multiple VFs. The end result is accelerated Container communication with segregation.

     

    References

     

    Prerequisite

    Follows the instructions in the following link to install Docker and VMA, HowTo Configure VMA Inside Docker Container (with MLNX OFED)

     

    Configuration

     

    Docker Compose

    Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a Compose file to configure your application’s services. Then, using a single command, you create and start all the services from your configuration, Compose is great for development, testing, and staging environments, as well as CI workflows.

     

    Using Compose is basically a three-step process.

    1. Define your app’s environment and install docker-compose.

    To install Docker compose use the following link  https://docs.docker.com/compose/install/

     

    2. Define the services that make up your app in docker-compose.yml so they can be run together in an isolated environment.

    To create docker compose yml and define the services https://docs.docker.com/compose/gettingstarted/

     

    Note: Creating app.py, requirements.txt and Dockerfile files is not important, you can skip it. You need just to create a directory for the project and to define services in a Compose file

     

    3. Lastly, run docker-compose up and Compose will start and run your app

    $ docker-compose up

     

    docker-compose.yml looks like this:

    version: '2'

    services:

    web:

        image: rhl7-vma-sockperf

        devices:

           - “/dev/infiniband/uverbs0:/dev/infiniband/uverbs0”

           - “/dev/infiniband/uverbs1:/dev/infiniband/uverbs1”

        volumes:

           - /tmp/mnt:/tmp/mny

        command: bash -c"timeout --signal=2 60 sockperf sr -i

           12.209.13.144 -p 9090 --tcp  > /tmp/mnt/run0"

        environment:

           - LD_PRELOAD=libvma.so

           - VMA_MEM_ALLOC_TYPE=1

           - VMA_STATS_FILE=/tmp/mnt/stat0

       Network_mode: “host”

        ulimits:

          memlock:

            soft: -1

            hard: -1

     

    To learn more about Docker Compose and how to install and run it, follow this link: https://docs.docker.com/compose/

     

    Docker-compose.yml

    The most important step is when starting docker compose is to create docker-compose.yml file that contains the services, image you work with, devices, VMA and sockperf commands.

    It is not mandatory to create other files such as ”app.py, requirements.txt and Dockerfile”.

    By looking to the previous yum example you will note the following:

    • Image: insert the image you already created for testing
    • Devices: share the devices “uverbs” you have in your machine
    • Command: insert sockperf command you want to run timeout --signal=2 60. It is used to interrupt the process after 60s
    • Environment: add VMA lib and VMA parameters

     

    Notes:

    • To use multi docker image you need to duplicate the previous block and change the web name to for example: web1, web2, web3 and change the command if needed.
    • To make sure that VMA traffic is offloaded in docker you should use VMA_STATS_FILE parameter in environments tag in yml file.
    • Docker-compose.yml is sensitive file, so any space or new line will cause it to fail.
    • After Creating docker-compose.yml file you can build and run your application with compose by
    $ docker-compose up

     

    VMA Command line inside docker-compose.yml

     

    The yml file contains Sockperf and VMA command under command and environment sections.

    command: bash -c"timeout --signal=2 60 taskset –c 7 sockperf sr -i 11.209.13.144 -p 12345 –msg-size 128 --tcp  > /tmp/mnt/run0"

        environment:

           - LD_PRELOAD=libvma.so

           - VMA_SPEC=latency

           - VMA_MEM_ALLOC_TYPE=1

           - VMA_STATS_FILE=/tmp/mnt/stat0

     

    command: bash –c "taskset –c 14 sockperf pp -i 11.209.13.144 -p 12345 –msg-size 128 –t 10  --tcp  > /tmp/mnt/run0"

        environment:

           - LD_PRELOAD=libvma.so

           - VMA_SPEC=latency

           - VMA_MEM_ALLOC_TYPE=1

           - VMA_STATS_FILE=/tmp/mnt/stat0

    So, the Full commands should look like this for both Server and Client :

    “VMA_MEM_ALLOC_TYPE=1 VMA_SPEC=latency LD_PRELOAD=libvma.so taskset –c 7 sockperf sr -i  11.209.13.144 -p 12345 –tcp”

    “VMA_MEM_ALLOC_TYPE=1 VMA_SPEC=latency LD_PRELOAD=libvma.so taskset –c 14 sockperf pp -i  11.209.13.144 -p 12345  -t 10 –tcp”

     

    See attached below for example.

     

    Configuring VFs

    The following link describes how to configure SR-IOV for ConnectX-4: HowTo Configure SR-IOV for ConnectX-4/ConnectX-5 with KVM (Ethernet)

     

    Performance Tuning

    • Run each containers on a separate core ( taskset –c <Core_Number> )

    • VMA_SPEC=latency

    • VMA_MEM_ALLOC=1

     

    See more about Performance Tuning of the server in Performance Tuning for Mellanox Adapters.

    And about VMA Performance tuning in VMA Performance Tuning Guide.