0 Replies Latest reply on Jan 18, 2017 7:39 PM by psanapathi

    Rate limiting Virtual Functions on ConnectX-4

    psanapathi

      Hi there!

       

      I've been trying to get the rate limit work with virtual functions on ConnectX-4. I have followed the steps given here - HowTo Configure Rate Limit per VF for ConnectX-4. I have two machines with ConnectX-4s, connected to each other, and each machine accommodates a couple of VMs. When I run 'iperf' among the VMs, the bandwidth is not being limited to the 'tx rate' I set. Any help would be greatly appreciated.

       

      $ lspci | grep Mellanox

      81:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

      81:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

      81:00.2 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]

      81:00.3 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]

      $ ibdev2netdev

      0000:81:00.0 mlx5_0 (MT4115 - MT1625K00420) CX456A - ConnectX-4 QSFP28 fw 12.17.1010 port 1 (ACTIVE) ==> eth1 (Up)

      0000:81:00.1 mlx5_1 (MT4115 - MT1625K00420) CX456A - ConnectX-4 QSFP28 fw 12.17.1010 port 1 (DOWN  ) ==> eth2 (Down)

      0000:81:00.2 mlx5_2 (MT4116 - MT1625K00420) CX456A - ConnectX-4 QSFP28 fw 12.17.1010 port 1 (DOWN  ) ==> eth0 (Down)

      0000:81:00.3 mlx5_3 (MT4116 - MT1625K00420) CX456A - ConnectX-4 QSFP28 fw 12.17.1010 port 1 (DOWN  ) ==> eth3 (Down)

      $ ip link show eth1 # after assigning MAC addresses

      5: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000

          link/ether 24:8a:07:27:d4:0e brd ff:ff:ff:ff:ff:ff

          vf 0 MAC 24:8a:07:17:00:00, spoof checking off, link-state auto

          vf 1 MAC 24:8a:07:17:00:01, spoof checking off, link-state auto

      $ ip link show eth1 # after setting rate limits

      5: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000

          link/ether 24:8a:07:27:d4:0e brd ff:ff:ff:ff:ff:ff

          vf 0 MAC 24:8a:07:17:00:00, tx rate 10000 (Mbps), spoof checking off, link-state auto

          vf 1 MAC 24:8a:07:17:00:01, tx rate 10000 (Mbps), spoof checking off, link-state auto

      $ cat /sys/class/net/eth1/device/sriov/0/config

      VF         : 0

      MAC        : 24:8a:07:17:00:00

      VLAN       : 0

      QoS        : 0

      SpoofCheck : OFF

      Trust      : OFF

      LinkState  : Follow

      MaxTxRate  : 10000

      $ cat /sys/class/net/eth1/device/sriov/1/config

      VF         : 1

      MAC        : 24:8a:07:17:00:01

      VLAN       : 0

      QoS        : 0

      SpoofCheck : OFF

      Trust      : OFF

      LinkState  : Follow

      MaxTxRate  : 10000