HowTo Configure QoS over SR-IOV

Version 2

    This post discuss QoS configuration for hosts using when configured with SR-IOV (VGT or VST modes).

    The testing scenarios were split to two QoS over VGT and QoS over VST. RoCE is out of the scope of this post.

    References

     

    Setup

    Two servers equipped with Mellanox ConnectX-3 adapter card, connected via SX1036 switch, each one is configured with several VMs.

     

    vms.png

     

    Switch Configuration

    1. All VLANs are allowed on all ports.

    2. PFC is enabled (for QoS)

    3. ETS is enabled (25% for each TC)

    switch (config)# vlan 1-10

    switch (config vlan 1-10) # exit

     

    switch (config)# interface ethernet 1/1 switchport hybrid allowed-vlan all

    switch (config)# interface ethernet 1/2 switchport hybrid allowed-vlan all

    switch (config)# interface ethernet 1/3 switchport hybrid allowed-vlan all

    switch (config)# interface ethernet 1/4 switchport hybrid allowed-vlan all

    switch (config)# dcb ets enable

    switch (config)# dcb ets tc bandwidth 25 25 25 25

    switch (config)# dcb priority-flow-control enable force

    Server Configuration

    The hypervisor configuration for QoS was done while playing around the mlnx_qos and vconfig commands.

    more information about the mlnx_qos command can be found in MLNX_OFED User Manual and in End-to-End QoS Configuration for Mellanox Switches (SwitchX) and Adapters.

    Remember, the mlnx_qos configuration is done on the hypervisor only.

     

    vconfig command was used to create VLANs on the VMs and apply the egress priority mapping on top.

    For example, for create a VLAN and add egress priority 7 to the packet use the following set of commands:

    # vconfig add eth1 7

    # ifconfig eth1.7 1.1.7.3/24 up

    # for i in {0..7}; do vconfig set_egress_map eth1.7 $i 7 ; done

    Testing method

    There are several options to flood the link, in this case we used iperf commands as follows

    invoke iperf server on one of the VMs with the following parameters

    #iperf -s -w 2m

    invoke iperf client on other VMs with the following parameters:

    #iperf -c <server's IP> -t 30 -i 2 -P8

     

     

    VGT Testing Scenarios

    1. Two VLANs, Strict Priority, No rate limiting over VGT

     

    VLAN Configuration: 2 VLANs: 3 and 5 are configured on each VM

    QoS Configuration: VLAN 3 is mapped to TC 3, VLAN 5 is mapped to TC 5, All TCs are configured as strict priority with no rate limiting. Remember - The QoS configuration is done on the hypervisor and not on the VM.

    #mlnx_qos -i eth1 -p 0,0,0,3,0,5,0,0 -s strict,strict,strict,strict,strict,strict,strict,strict -r 0,0,0,0,0,0,0,0

     

    Testing - running iperf on each VLAN (two streams)

    Results: as expected, The traffic on VLAN 3 pass the switch but did not arrive to the CPU. Only traffic on VLAN 5 will reach the CPU.

     

    2. Two VLANs, Strict Priority, rate limiting over VGT

    VLAN Configuration: 2 VLANs: 3 and 5 are configured on each VM

    QoS Configuration: VLAN 3 is mapped to TC 3, VLAN 5 is mapped to TC 5, All TCs are configured as strict priority with rate limiting of 2Gb/s on TC 5. The idea is to supply strict priority to certain level, but not more than that.

     

    #mlnx_qos -i eth1 -p 0,0,0,3,0,5,0,0 -s strict,strict,strict,strict,strict,strict,strict,strict -r 0,0,0,0,0,2,0,0

     

    Testing - running iperf on each VLAN (two streams)

    Results: as expected, Traffic on VLAN 5 reached the CPU up to 3Gb/s. Traffic on VLAN 3 reached as well with the rest of the port bandwidth.

     

    3. Two VLANs, ETS, no rate limiting over VGT

    VLAN Configuration: 2 VLANs: 3 and 5 are configured on each VM

    QoS Configuration: VLAN 3 is mapped to TC 3, VLAN 5 is mapped to TC 5, All TCs are configured as ETS when VLAN 3 gets 10% and VLAN 5 gets 90%. no rate limiting was configured.

     

    #mlnx_qos -i eth1 -p 0,0,0,3,0,5,0,0 -s ets,ets,ets,ets,ets,ets,ets,ets,-t 0,0,0,10,0,90,0,0 -r 0,0,0,0,0,0,0,0

     

    Testing - running iperf on each VLAN (two streams)

    Results: as expected, Traffic on VLAN 5 reached the CPU with 90% of the port capacity while Traffic on VLAN 3 reached as well with 10% of the port capacity.

     

    4. Three VLANs, ETS and strict priority, rate limiting over VGT

    VLAN Configuration: 3 VLANs: 3, 5 and 7 are configured on each VM.

    QoS Configuration: VLAN 3 is mapped to TC 3, VLAN 5 is mapped to TC 5, VLAN7 is mapped to TC 7. TC 3 and 5 are configured as ETS, while TC 7 is configured in Strict Priority. VLAN 3 gets 10% and VLAN 5 gets 90%. rate limiting was configured only on VLAN 7.

     

    #mlnx_qos -i eth1 -p 0,0,0,3,0,5,0,7 -s ets,ets,ets,ets,ets,ets,ets,strict,-t 0,0,0,10,0,90,0,0 -r 0,0,0,0,0,0,0,1

     

    Testing - running iperf on each VLAN (three streams)

    Results: as expected, Traffic on VLAN 7 reached the CPU 1Gb/s while Traffic on VLAN 3 reached 10% of the rest of the bandwidth and VLAN 5 reached to 90%.

     

    5. Three VLANs, ETS and strict priority, rate limiting over VGT

    VLAN Configuration: 2 VLANs: 3 and 5 are configured on each VM. VLAN 7 is configured on the hypervisors.

    QoS Configuration: VLAN 3 is mapped to TC 3, VLAN 5 is mapped to TC 5, VLAN7 is mapped to TC 7. TC 3 and 5 are configured as ETS, while TC 7 is configured in Strict Priority. VLAN 3 gets 10% and VLAN 5 gets 90%. rate limiting was configured only on VLAN 7.

     

    #mlnx_qos -i eth1 -p 0,0,0,3,0,5,0,7 -s ets,ets,ets,ets,ets,ets,ets,strict,-t 0,0,0,10,0,90,0,0 -r 0,0,0,0,0,0,0,1

     

    Testing - running iperf on each VLAN (three streams)

    Results: as expected, Traffic on VLAN 7 reached the CPU 1Gb/s while Traffic on VLAN 3 reached 10% of the rest of the bandwidth and VLAN 5 reached to 90%.

     

    VST Testing Scenarios

    Two VMs each connected with one VF (untagged) and the hypervisor performs the VLAN tagging and priority (QoS) mapping via VST mode.

    For example:

    #ip link set eth1 vf 1 vlan 3 qos 3

    #ip link set eth1 vf 2 vlan 5 qos 5

     

    Running mlnx_qos command on the hypervisor in similar manner to VGT tests.

    Results: as expected (similar to the VGT results)