HowTo Enable/Disable RoCE on VFs (ConnectX-4)

Version 4

    This post shows how to Enable/Disable RoCE on ConnectX-4 VFs.

    The procedure here supported starting from MLNX_OFED 4.0 (minimum firmware 12.18.0248) on ConnectX-4.

     

    References

     

    Overview

    By default, when configuring several VFs on the hypervisor, all VFs will be enabled with RoCE. This means that they require more OS memory comparing to Ethernet only VFs. In case you are only interested in Ethernet (no RDMA) on the VF, and you wish to save the hypervisor memory, you can disable RoCE on the VF from the hypervisor. By doing this, the VF will request less host memory from hypervisor.

     

    Note: The VF driver in virtual machine must be reloaded for the setting to take effect.

    Note: The setting is not persistent. When PF driver is reloaded, the RoCE feature on virtual function is enabled back by default.

     

    Configuration

    1. To query the RoCE feature support of all the virtual functions, read the vf_roce fiel from /sys/class/net/if_name/settings from the hypervisor. The if_name is the netdev interface name of the physical function.

     

    In the example below, the physical function’s netdev named ens1f0 with 8 VFs (the firmware ini SRIOV settings are SRIOV_EN=1 NUM_OF_VFS=8).

    RoCE is disabled (0) on VFs  0, 4, and 7 and enabled (1) on VFS 1, 2, 3, 5, and 6.

    # cat /sys/class/net/ens1f0/settings/vf_roce

    vf_num 0: 0

    vf_num 1: 1

    vf_num 2: 1

    vf_num 3: 1

    vf_num 4: 0

    vf_num 5: 1

    vf_num 6: 1

    vf_num 7: 0

     

    2. To enable/disable RoCE on virtual function ,  write “<vf_num> {1 | 0}” to the vf_roce file.

     

    For example, to enable RoCE on virtual function 4, run:

    # echo 4 1 > /sys/class/net/ens1f0/settings/vf_roce

     

    # cat /sys/class/net/ens1f0/settings/vf_roce

    vf_num 0: 0

    vf_num 1: 1

    vf_num 2: 1

    vf_num 3: 1

    vf_num 4: 1

    vf_num 5: 1

    vf_num 6: 1

    vf_num 7: 0

     

    To disable RoCE on VF 2 run:

    # echo 2 0 > /sys/class/net/ens1f0/settings/vf_roce

     

    # cat /sys/class/net/ens1f0/settings/vf_roce

    vf_num 0: 0

    vf_num 1: 1

    vf_num 2: 0

    vf_num 3: 1

    vf_num 4: 1

    vf_num 5: 1

    vf_num 6: 1

    vf_num 7: 0

     

    3. Reload the driver on the VM.