HowTo Configure and Probe VFs on mlx5 Drivers

Version 6

    Probing Virtual Functions (VFs) after SR-IOV is enabled is not always a good idea. It might be very inconvenient if there are a lot of VFs created and probed in the hypervisor. Not probing VFs can save host side resource usage by VF instances, which would be eventually probed to the VMs.

    Note: You can opt to probe VFs by drivers, but that depends on which kernel versions are installed. Use one of the following methods:

    1. If the kernel version is 4.12 or above, we recommend using the PCI sysfs interface sriov_drivers_autoprobe. For more information see The linux-next branch.

    2. If kernel version is older than rev. 4.12, use mlx5_core module parameter probe_vf and with MLNX_OFED rev. 4.1.

     

    This document focuses on the second option.

     

     

    References

     

    Configuration

    The probe_vf mlx5_core module parameter is exposed at:

    /sys/module/mlx5_core/parameters/probe_vf

    where:

    • probe_vf=0 indicates VFs will not bind to mlx5_core when SR-IOV is enabled.
    • probe_vf=1 indicates VFs will bind to mlx5_core and net device will be created when VFs are created. This is the default value.

     

    To disable probe_vf, run:

    # echo 0 > /sys/module/mlx5_core/parameters/probe_vf

     

    To enable probe_vf, run:

    # echo 1 > /sys/module/mlx5_core/parameters/probe_vf

     

    To make use of this feature, you must make the change before enabling SR-IOV. Making changes to this entry will not effect VFs that are already enabled. It will take effect after disabling SR-IOV, and enabling it again.

    Note: The bind/unbind sysfs for the mlx5_core can be affected by this module parameter. For example, if probe_vf is set as 0 but you try to bind VFs to mlx5_core, it will not work.

     

     

     

    Verification

    Disable VF Probing

    1. As an example, if you do not want VFs probed when enabling SR-IOV, run:

    # echo 0 > /sys/module/mlx5_core/parameters/probe_vf

     

    2. After this, enable SR-IOV by running:

    # echo 2 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs

     

    Verify from the PCI bus:

    # lspci | grep Mellanox

    81:00.0 Ethernet controller: Mellanox Technologies MT27620 Family

    81:00.1 Ethernet controller: Mellanox Technologies MT27620 Family [Virtual Function]

    81:00.2 Ethernet controller: Mellanox Technologies MT27620 Family [Virtual Function]

     

    3. Since we choose not to probe SR-IOV, the InfiniBand/Ethernet interface should not be created for the VFs.

    # ibdev2netdev

    mlx5_0 port 1 ==> p7p1 (Down)

    mlx5_1 port 1 ==> enp6s0f1 (Down)

     

    4. In addition, if you use the dmesg command to write kernel messages to standard output, it should display the following message:

    #dmesg

    ….

    mlx5_core 0000: 81:00.0: VFs are not binded to mlx5_core

    ….

     

    Enable VF Probing

    This is the default configuration, however, if probing was disabled, to re-enable it run the following:

     

    1. If you want to probe VFs on the host side, first make sure SR-IOV is disabled.

    # echo 0 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs

     

    2. Then, enable probe VFs as follows:

    # echo 1 > /sys/module/mlx5_core/parameters/probe_vf

     

    3. Enable SR-IOV again:

    # echo 2 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs

     

    4. Verify your changes from the PCI bus:

    # lspci | grep Mellanox

    81:00.0 Ethernet controller: Mellanox Technologies MT27620 Family

    81:00.1 Ethernet controller: Mellanox Technologies MT27620 Family [Virtual Function]

    81:00.2 Ethernet controller: Mellanox Technologies MT27620 Family [Virtual Function]

     

    5. Verify VFs are probed by mlx5 driver. In this case, you will see two more interfaces, mlx5_2 and mlx5_3.

    # ibdev2netdev

    mlx5_0 port 1 ==> p7p1 (Down)

    mlx5_1 port 1 ==> enp6s0f1 (Down)

    mlx5_2 port 1 ==> p7p1_1(Down)

    mlx5_3 port 1 ==> p7p1_2 (Down)