HowTo Configure probe VFs on mlx5 driver

Version 4

    Sometimes it is not desirable to probe the virtual functions after SR-IOV is enabled. This can save host side resource usage by VF instances which would be eventually probed to the VMs. It might be very inconvenient if there are a lot of VFs created and probed in the hypervisor.

     

    Note: There are two options to choose whether to probe VFs by driver, depends on kernel versions.

    1. if kernel version is 4.12 or above, It is recommended to use the PCI sysfs interface sriov_drivers_autoprobe. For more info see The linux-next branch.

    2. If kernel version is older than 4.12, use mlx5_core module parameter probe_vf and with MLNX_OFED 4.1.

     

    This document focuses on the second option.

     

     

    References

     

    Configuration

    The probe_vf mlx5_core module parameter is exposed at:

    /sys/module/mlx5_core/parameters/probe_vf

    while

    • probe_vf=0 means VFs will not bind to mlx5_core when sriov is enabled.
    • probe_vf=1 means VFs will bind to mlx5_core and net device will be created when VFs are created. This is the default value.

     

    To disable probe_vf run:

    # echo 0 > /sys/module/mlx5_core/parameters/probe_vf

     

    To enable probe_vf, run:

    # echo 1 > /sys/module/mlx5_core/parameters/probe_vf

     

    To make use of this feature, you must make the change before enabling SR-IOV. Making change of this entry will not effect VFs which are already enabled. It will take effect after disabling SR-IOV, and enabling again.

    Note: The bind/unbind sysfs for mlx5_core may be affected by this module parameter. For example, if probe_vf is set as 0, but you try to bind VFs to mlx5_core, it will not work.

     

     

     

    Verification

    Disable VFs probing

    1. As an example, if you don't want VFs probed when enabling SR-IOV. Run:

    # echo 0 > /sys/module/mlx5_core/parameters/probe_vf

     

    2. After this, enable SR-IOV by running:

    # echo 2 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs

     

    Verify from PCI bus:

    # lspci | grep Mellanox

    81:00.0 Ethernet controller: Mellanox Technologies MT27620 Family

    81:00.1 Ethernet controller: Mellanox Technologies MT27620 Family [Virtual Function]

    81:00.2 Ethernet controller: Mellanox Technologies MT27620 Family [Virtual Function]

     

    3. Since we choose not to probe SR-IOV, IB/Eth interface should not be created for the VFs.

    # ibdev2netdev

    mlx5_0 port 1 ==> p7p1 (Down)

    mlx5_1 port 1 ==> enp6s0f1 (Down)

     

    4. In addition, dmesg should show following message:

    #dmesg

    ….

    mlx5_core 0000: 81:00.0: VFs are not binded to mlx5_core

    ….

     

    Enable VFs probing

    This is the default configuration, however, if probing was disabled, to re-enable it run the following:

     

    1. In case you want to probe VFs in host side. First make sure SR-IOV is disabled

    # echo 0 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs

     

    2. Then, enable probe VFs by doing:

    # echo 1 > /sys/module/mlx5_core/parameters/probe_vf

     

    3. Enable SR-IOV again:

    # echo 2 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs

     

    4. Verify from PCI bus:

    # lspci | grep Mellanox

    81:00.0 Ethernet controller: Mellanox Technologies MT27620 Family

    81:00.1 Ethernet controller: Mellanox Technologies MT27620 Family [Virtual Function]

    81:00.2 Ethernet controller: Mellanox Technologies MT27620 Family [Virtual Function]

     

    5. Verify VFs are probed by mlx5 driver. In this case, you will see two more interfaces mlx5_2 and mlx5_3

    # ibdev2netdev

    mlx5_0 port 1 ==> p7p1 (Down)

    mlx5_1 port 1 ==> enp6s0f1 (Down)

    mlx5_2 port 1 ==> p7p1_1(Down)

    mlx5_3 port 1 ==> p7p1_2 (Down)