How-To Configure Mellanox VMA inside Docker Container using Docker Composer

Version 3

    This post describes how to install Docker container,  how to configure Mellanox VMA Kernel Bypass inside Docker, and how to create multiple Docker images with Multiple VFs. The end result is accelerated Container communication with segregation.





    Follows the instructions in the following link to install Docker and VMA, HowTo Configure VMA Inside Docker Container (with MLNX OFED)




    Docker Compose

    Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a Compose file to configure your application’s services. Then, using a single command, you create and start all the services from your configuration, Compose is great for development, testing, and staging environments, as well as CI workflows.


    Using Compose is basically a three-step process.

    1. Define your app’s environment and install docker-compose.

    To install Docker compose use the following link


    2. Define the services that make up your app in docker-compose.yml so they can be run together in an isolated environment.

    To create docker compose yml and define the services


    Note: Creating, requirements.txt and Dockerfile files is not important, you can skip it. You need just to create a directory for the project and to define services in a Compose file


    3. Lastly, run docker-compose up and Compose will start and run your app

    $ docker-compose up


    docker-compose.yml looks like this:

    version: '2'



        image: rhl7-vma-sockperf


           - “/dev/infiniband/uverbs0:/dev/infiniband/uverbs0”

           - “/dev/infiniband/uverbs1:/dev/infiniband/uverbs1”


           - /tmp/mnt:/tmp/mny

        command: bash -c"timeout --signal=2 60 sockperf sr -i

  -p 9090 --tcp  > /tmp/mnt/run0"



           - VMA_MEM_ALLOC_TYPE=1

           - VMA_STATS_FILE=/tmp/mnt/stat0

       Network_mode: “host”



            soft: -1

            hard: -1


    To learn more about Docker Compose and how to install and run it, follow this link:



    The most important step is when starting docker compose is to create docker-compose.yml file that contains the services, image you work with, devices, VMA and sockperf commands.

    It is not mandatory to create other files such as ”, requirements.txt and Dockerfile”.

    By looking to the previous yum example you will note the following:

    • Image: insert the image you already created for testing
    • Devices: share the devices “uverbs” you have in your machine
    • Command: insert sockperf command you want to run timeout --signal=2 60. It is used to interrupt the process after 60s
    • Environment: add VMA lib and VMA parameters



    • To use multi docker image you need to duplicate the previous block and change the web name to for example: web1, web2, web3 and change the command if needed.
    • To make sure that VMA traffic is offloaded in docker you should use VMA_STATS_FILE parameter in environments tag in yml file.
    • Docker-compose.yml is sensitive file, so any space or new line will cause it to fail.
    • After Creating docker-compose.yml file you can build and run your application with compose by
    $ docker-compose up


    VMA Command line inside docker-compose.yml


    The yml file contains Sockperf and VMA command under command and environment sections.

    command: bash -c"timeout --signal=2 60 taskset –c 7 sockperf sr -i -p 12345 –msg-size 128 --tcp  > /tmp/mnt/run0"



           - VMA_SPEC=latency

           - VMA_MEM_ALLOC_TYPE=1

           - VMA_STATS_FILE=/tmp/mnt/stat0


    command: bash –c "taskset –c 14 sockperf pp -i -p 12345 –msg-size 128 –t 10  --tcp  > /tmp/mnt/run0"



           - VMA_SPEC=latency

           - VMA_MEM_ALLOC_TYPE=1

           - VMA_STATS_FILE=/tmp/mnt/stat0

    So, the Full commands should look like this for both Server and Client :

    “VMA_MEM_ALLOC_TYPE=1 VMA_SPEC=latency taskset –c 7 sockperf sr -i -p 12345 –tcp”

    “VMA_MEM_ALLOC_TYPE=1 VMA_SPEC=latency taskset –c 14 sockperf pp -i -p 12345  -t 10 –tcp”


    See attached below for example.


    Configuring VFs

    The following link describes how to configure SR-IOV for ConnectX-4: Docker RDMA SRIOV Networking with ConnectX4/ConnectX5


    Performance Tuning

    • Run each containers on a separate core ( taskset –c <Core_Number> )

    • VMA_SPEC=latency



    See more about Performance Tuning of the server in Performance Tuning for Mellanox Adapters.

    And about VMA Performance tuning in VMA Performance Tuning Guide.