This post describes how to install Docker container, how to configure Mellanox VMA Kernel Bypass inside Docker, and how to create multiple Docker images with Multiple VFs. The end result is accelerated Container communication with segregation.
Follows the instructions in the following link to install Docker and VMA, HowTo Configure VMA Inside Docker Container (with MLNX OFED)
Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a Compose file to configure your application’s services. Then, using a single command, you create and start all the services from your configuration, Compose is great for development, testing, and staging environments, as well as CI workflows.
Using Compose is basically a three-step process.
1. Define your app’s environment and install docker-compose.
To install Docker compose use the following link https://docs.docker.com/compose/install/
2. Define the services that make up your app in docker-compose.yml so they can be run together in an isolated environment.
To create docker compose yml and define the services https://docs.docker.com/compose/gettingstarted/
Note: Creating app.py, requirements.txt and Dockerfile files is not important, you can skip it. You need just to create a directory for the project and to define services in a Compose file
3. Lastly, run docker-compose up and Compose will start and run your app
$ docker-compose up
docker-compose.yml looks like this:
command: bash -c"timeout --signal=2 60 sockperf sr -i
126.96.36.199 -p 9090 --tcp > /tmp/mnt/run0"
To learn more about Docker Compose and how to install and run it, follow this link: https://docs.docker.com/compose/
The most important step is when starting docker compose is to create docker-compose.yml file that contains the services, image you work with, devices, VMA and sockperf commands.
It is not mandatory to create other files such as ”app.py, requirements.txt and Dockerfile”.
By looking to the previous yum example you will note the following:
- Image: insert the image you already created for testing
- Devices: share the devices “uverbs” you have in your machine
- Command: insert sockperf command you want to run timeout --signal=2 60. It is used to interrupt the process after 60s
- Environment: add VMA lib and VMA parameters
- To use multi docker image you need to duplicate the previous block and change the web name to for example: web1, web2, web3 and change the command if needed.
- To make sure that VMA traffic is offloaded in docker you should use VMA_STATS_FILE parameter in environments tag in yml file.
- Docker-compose.yml is sensitive file, so any space or new line will cause it to fail.
- After Creating docker-compose.yml file you can build and run your application with compose by
$ docker-compose up
VMA Command line inside docker-compose.yml
The yml file contains Sockperf and VMA command under command and environment sections.
command: bash -c"timeout --signal=2 60 taskset –c 7 sockperf sr -i 188.8.131.52 -p 12345 –msg-size 128 --tcp > /tmp/mnt/run0"
command: bash –c "taskset –c 14 sockperf pp -i 184.108.40.206 -p 12345 –msg-size 128 –t 10 --tcp > /tmp/mnt/run0"
So, the Full commands should look like this for both Server and Client :
“VMA_MEM_ALLOC_TYPE=1 VMA_SPEC=latency LD_PRELOAD=libvma.so taskset –c 7 sockperf sr -i 220.127.116.11 -p 12345 –tcp”
“VMA_MEM_ALLOC_TYPE=1 VMA_SPEC=latency LD_PRELOAD=libvma.so taskset –c 14 sockperf pp -i 18.104.22.168 -p 12345 -t 10 –tcp”
See attached below for example.
The following link describes how to configure SR-IOV for ConnectX-4: Docker RDMA SRIOV Networking with ConnectX4/ConnectX5
Run each containers on a separate core ( taskset –c <Core_Number> )
See more about Performance Tuning of the server in Performance Tuning for Mellanox Adapters.
And about VMA Performance tuning in VMA Performance Tuning Guide.