This post supplies basic configuration and troubleshooting information for using OpenStack with ConnectX-4 (Ethernet or InfiniBand).
- HowTo Configure SR-IOV for ConnectX-4/ConnectX-5 with KVM (Ethernet)
- HowTo Configure SR-IOV for Connect-IB/ConnectX-4 with KVM (InfiniBand)
1. Follow HowTo Configure SR-IOV for ConnectX-4/ConnectX-5 with KVM (Ethernet) or HowTo Configure SR-IOV for Connect-IB/ConnectX-4 with KVM (InfiniBand) to enable SR-IOV on the server (Ethernet or InfiniBand).
3. Enable eIPoIB module via the configuration file /etc/infiniband/openib.conf:
eIPoIB is used in OpenStack on the network node (that provides the DHCP server). You do not need to enable that on the compute nodes (hypervisors).
For more information, please refer to eIPoIB configuration in Mellanox OFED User Manual.
1. Ensure that the BIOS is enabled for SR-IOV, otherwise you might get the following error.
echo 4 > /sys/class/infiniband/mlx5_0/device/sriov_numvfs
bash: echo: write error: Cannot allocate memory
For example, enter the server BIOS, make sure “Intel HT technology” is disabled, and enable the “SR-IOV PCI” option located under I/O options or CPU options.
Each server can have a different BIOS configuration. Refer also to your specific BIOS documentation.
2. If you can not see ConnextX-4 in lspci, run the following to update the pci information.
# sudo update-pciids