This post shows how to use single Mellanox ConnectX-4/ConnectX-5 InfiniBand HCA in Kubernetes cluster shared among multiple Pods.
You must use Kubernetes version 1.10.3 or higher.
One RDMA device (HCA) can be shared among multiple Pods running in a Kubernetes worker node. In this mode virtual networking device support is provided by networking connectivity for IP networking using vxlan or veth networking devices.
Configuration and setup involves following steps.
- HCA device plugin configuration and installation
- Create device plugin configuration
- Install device plugin
- Pod configuration
- Check configuration
- Check device plugin configuration
- Check sriov cni installation
1.HCA device plugin configuration and installation
Apply HCA device plugin configuration
This applies HCA device plugin configuration as Kubernetes ConfigMap.
2.Deploy HCA device plugin
3. Pod Configuration
Each Pod’s container configuration needs to have configuration to indicate that it demands one virtual networking device.
Example Pod configuration can be found below.
4. Check configuration
Check device plugin and configuration
#kubectl get ds --namespace=kube-system