This post describes how to create and run a docker containers which supports RDMA, using CentOS 7.0 as the OS of the host and the container.
For Docker installation per OS see: https://docs.docker.com/engine/installation
This site has all the relevant info we need for a quick start. See docker fundamentals tab as well.
In addition, there are few nice videos with demos see: https://training.docker.com/self-paced-training
As of time being, performing RDMA from containers requires some amount of manual configuration.
The setup assumes two hosts connected via a switch or any L2 domain. In this example, the subnet of 192.168.101.xxx is used.
Configure both hosts to be enabled with RoCE and the drivers shipped with CentOS 7.
Refer to this post HowTo Setup RDMA Connection using Inbox Driver (RHEL, Ubuntu)
Once there is RoCE connectivity between the hosts. we are ready to launch a container which will support RDMA operations.
For this, we will need either to make the container privileged, or install a git revision of docker, that supports exposing to the container specific devices.
To get , compile docker and lunch it on CentOS 7, follow the next steps:
1. Get the docker sources
# git clone https://github.com/docker/docker.git
2. Install the required dependencies for building docker
# yum install -y golang golang-github-godbus-dbus-devel golang-github-coreos-go-systemd-devel golang-github-gorilla-mux-devel golang-github-gorilla-context-devel golang-googlecode-net-devel golang-googlecode-sqlite-devel golang-github-syndtr-gocapability-devel golang-github-kr-pty-devel
# yum install -y sqlite-devel device-mapper-devel device-mapper-event-devel glibc-static btrfs-progs-devel device-mapper-devel systemd-devel
AUTO_GOPATH=1 ./hack/make.sh dynbinary
4. Stop system docker service:
# service docker stop
5. Launch docker service:
# cd bundles/1.2.0-dev/dynbinary
# ./docker -d --selinux-enabled &
Using the docker you just compiled, we can launch a centos container, that is allowed to access the infiniband devices uverbs0 and the RDMA-CM. Due to RDMA-CM limitations, the container must use the host network name space. The following command line launches such container:
# ./docker run --net=host --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm \
-t -i centos /bin/bash
Now, we need to install the user space RDMA software in the container:
bash-4.2 # yum install -y libibverbs-utils libibverbs-devel libibverbs-devel-static libmlx4 \
libmlx5 ibutils libibcm libibcommon libibmad libibumad
bash-4.2 # yum install -y rdma librdmacm-utils librdmacm-devel librdmacm libibumad-devel perftest
And finally, we can test RDMA traffic from the container, using the same commands as above (ib_send_bw).