This post shows how to use Mellanox ConnectX-4/ConnectX-5 RoCE RoCE HCAs with Docker Containers.
RDMA Support to Docker containers with single RDMA device is provided using light weight MACVLAN networking devices using Mellanox ConnectX-4/ConnectX-5 RoCE HCAs.
This is helpful when running in Docker engine in bare metal system or using HCA SR-IOV VF inside a VM and running Docker containers inside a VM.
This provides a very simple and quick way to start using RoCE HCAs in containers.
Configuration and setup involves following steps.
- Create one or more tenant networks
- Start Containers
Note: SR-IOV is optional in above setup/configuration. RoCE with macvlan can be directly use with PF.
However when using with combination of SR-IOV VFs and MACVLAN, it can provide few benefits.
Such as (a) transparent vlan offload (b) larger scale of containers per system.
1. Create one or more tenant networks
#docker network create -d macvlan --subnet=220.127.116.11/24 --gateway=18.104.22.168 -o parent=ens2f0 mynet
Here ens2f0 is a netdevice of the RoCE HCA. Change it to right name based on your system configuration.
Above configuration doesn't uses vlan.
In case if you want to use use vlan based isolated networks, a VLAN ID can be provided and tenant network can be created as below.
#docker network create -d macvlan --subnet=22.214.171.124/24 --gateway=126.96.36.199 -o parent=ens2f0.50 mynet
Here ens2f0 is a netdevice of the RoCE HCA and 50 is the VLAN ID. Change it to right name based on your system configuration.
2. Run a container with RDMA devices and macvlan devices
In this mode all containers running on a host shares the single RDMA device. However each container can get to see only GID table entries which belongs to its own network namespace and its own MACVLAN netdevice.
#docker run --net=mynet --device=/dev/infiniband -it mellanox/mlnx_ofed_linux-4.4-188.8.131.52-centos7.4 bash
Sample Dockerfile to build Mellanox OFED based on Centos or Ubuntu base image can be found in below git repository.