HowTo Configure SR-IOV VFs on Different ConnectX-3 Ports

Version 9

    This post describes various SR-IOV VF configuration options used over ConnectX-3 adapters. This post should be in Openstack SR-IOV solution (starting from OpenStack Juno release).

     

    References

     

    Prerequisites

    • MLNX_OFED 2.3 or higher on compute nodes


    Configuring SR-IOV

    In the host BIOS: Make sure SR-IOV is enabled.

    In boot loader: In grub configuration ( e.g. /boot/grub/grub.cfg) make sure the line of the kernel has 'intel_iommu=pt' or 'intel_iommu=on'.

     

    For Example:

    linux   /boot/vmlinuz-3.13.0-32-generic root=UUID=766c6f7c-1ead-4a71-b382-4d75bf904101 ro recovery nomodeset BOOTIF=01-00-25-90-e3-a3-cc intel_iommu=on

          

    Configuring 8 VFs on a single port NIC while all VFs are probed

    In /etc/modprobe/mlx4.conf edit the following:

    options mlx4_core port_type_array=2 num_vfs=8 probe_vf=8  log_num_mgm_entry_size=-1

    Configuring 8 VFs on a dual port NIC while all VFs are probed on port 1

    In /etc/modprobe.d/mlx4.conf edit the following:

    options mlx4_core port_type_array=2,2 num_vfs=8,0,0 probe_vf=8,0,0  log_num_mgm_entry_size=-1

     

    Configuring 8 VFs on a dual port NIC while 4 VFs are probed on port 1 and 4 VFs are probed on port 2

    In /etc/modprobe.d/mlx4.conf edit the following:

    options mlx4_core port_type_array=2,2 num_vfs=4,4,0 probe_vf=4,4,0  log_num_mgm_entry_size=-1

     

    Configuring 8 VFs on a dual port NIC while 2 VFs are probed on port 1 and 2 VFs are probed on port 2 and 4 VFs are probed on both ports

    In /etc/modprobe.d/mlx4.conf edit the following:

    options mlx4_core port_type_array=2,2 num_vfs=2,2,4 probe_vf=2,2,4  log_num_mgm_entry_size=-1

       

     

    All those examples assumes single adapter installed on the host. Refer to the  MLNX_OFED User Manual for additional examples and more detailed explanations.

     

    Note: In case of OpenStack Juno release, all the VFs that will be allocated to VMs must be probed VFs.

     

    Restart openibd before Installing Openstack on the compute nodes

    #/etc/init.d/openidb restart