HowTo Configure Rate Limit per VF for ConnectX-4/ConnectX-5

Version 11

    This post discusses and presents the configuration for setting rate limits per Virtual Function (VF) using ConnectX-4/ConnectX-5 adapter cards families.

    The max rate limit is supported starting from MLNX_OFED v3.4, while the min rate limit is supported starting from MLNX_OFED 4.0 and later.

     

    References

     

    Configuration

    As a prerequisite, follow HowTo Configure SR-IOV for ConnectX-4 with KVM (Ethernet). Make sure the Virtual Machine (VM) is configured with a MAC address.

    In the following example, the administrator assigned this MAC address 00:22:33:44:55:66 to the VM located on VF-0.

    # ip link show

    ...

    9: ens785f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT qlen 1000

        link/ether e4:1d:2d:f2:a4:89 brd ff:ff:ff:ff:ff:ff

        vf 0 MAC 00:22:33:44:55:66, spoof checking off, link-state auto

        vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
        vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto    vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto...

     

    There are two options to set rate limit per VF:

     

    1. Using the command ip link set.

    2. Using the sysfs (In case MLNX_OFED 4.0 is installed).

     

    See examples below:

    Use the standard ip link command (supported in Kernel version 3.10 and later)

     

    Format:

    # ip link set <PF_IF_NAME> vf <VF_IDX> max_tx_rate  <MAX_RATE_IN_MBIT/S> min_tx_rate <MIN_RATE_IN_MBIT/S>

     

    Example to enable rate limiting of 1000Mb/s and minimum of 10Mb/s:

    # ip link set ens785f1 vf 0 max_tx_rate 1000 min_tx_rate 10

     

    The rate is measured in Mb/s.

     

    To disable or stop the rate limit per VF, set the rate to 0.

    # ip link set ens785f1 vf 0 max_tx_rate 0 min_tx_rate 0

     

    Note: An older version of iproute2 may have only a rate option (this is equal to the max rate)

    ip link set <interface_name> vf <vf_id> rate <value>

     

    For more ip link examples, refer to  HowTo Set Virtual Network Attributes on a Virtual Function (SR-IOV).

    To verify, run ip link show command.

     

    Rate limit is Enabled (Max + Min).

     

    # ip link show

    ...

    9: ens785f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT qlen 1000

        link/ether e4:1d:2d:f2:a4:89 brd ff:ff:ff:ff:ff:ff

        vf 0 MAC 00:22:33:44:55:66, tx rate 1200 (Mbps), max_tx_rate 1200Mbps, min_tx_rate 1000Mbps spoof checking off, link-state auto

        vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto

        vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto

        vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto

     

    sysfs (Option 2)

    Run the echo command with the file located under

    • /sys/class/net/<ETH_IF_NAME> /device/sriov/<VF index>/max_tx_rate
    • /sys/class/net/<ETH_IF_NAME> /device/sriov/<VF index>/min_tx_rate

     

    For example, to set the maximum rate to 1Gb/s (1000Mb/s) and the minimum rate to 10Mb/s, run:

    # echo 1000 > /sys/class/net/ens785f1/vf/0/max_tx_rate

    # echo 10  > /sys/class/net/ens785f1/vf/0/min_tx_rate

     

    Note: This configuration is not persistent (it resets with driver restart).

     

    Once you enable the rate limit, test the speed rate from the VM and see that the traffic is limited to the rate you configured.

     

    To get the sysfs configuration, run:

    # cat /sys/class/net/ens785f1/device/sriov/0/config

    VF : 0

    MAC : 00:22:33:44:55:66

    VLAN : 0

    QoS : 0

    SpoofCheck : OFF

    Trust : OFF

    LinkState : Follow

    MinTxRate : 10

    MaxTxRate : 1000