Although it might sound trivial, not all 3rd-party switch ports are the same. According to the latest Tolly report we found "unfairness" (defined as "unfair bandwidth and distribution in test cases") between a group of ports in switches based on the Broadcom StrataXGS Tomahawk (Tomahawk) switch silicon. This post explains the results of the Tolly Report and describes how we ran tests on the switch to identify fairness using IXIA, Spirent's test equipment, or simple iperf tool. This test can be used for simple benchmark to compare fairness between the switch ports.
- What is Fairness?
- What is the Tomahawk-Based Switch Port Distribution Problem?
- Will I Find the Same Behavior in all Tomahawk-Based Switches?
- How Can I Demonstrate the Unfairness Problem?
- How Can I Reproduce the Tolly Unfairness Results?
- What Testing Tool Can I Use?
- What Contributes to Unfairness?
- Can This Problem Be Solved in Software for Tomahawk IC?
- Why Don't We See This Behavior in Spectrum IC?
What is Fairness?
Fairness is an important property of any interconnection network. Different traffic flows, having the same priority, should all get equal access to shared resources in the network. When one traffic flow receives more resources (for example, buffer allocation) than other traffic flow, unfairness is introduced.
What is the Tomahawk-Based Switch Port Distribution Problem?
The problem arises when 32 ports are divided to two groups of 16 ports each. Each group has its data path and different buffer.
This leads to unfairness in tests when traffic is being sent from one group to the other when in one group you have different number of ports in each group.
Will I Find the Same Behavior in all Tomahawk-Based Switches?
Yes, but it might be in a different port group, as it depends on the switch board's design and wiring.
How Can I Demonstrate the Unfairness Problem?
You can start by sending traffic from any amount of ports of the switch in line rate to an egress port.
If, for example, one port is mapped to the one buffer while all the other ports will be mapped to the second buffer, you will see how the unfairness condition occurs.
In the example below taken from the tolly report, it is easy to see that in the switch that was used here, Ports 9-24 were mapped to one buffer, while Ports 1-8 and 25-32 were mapped to another buffer.
In test 1, all ports in the test where part of the same buffer, in this case the benchmarks showed similar results comparing to Spectrum switch. All ports used the same shared buffer.
By comparison, test 5, one port (port 8) is mapped to one buffer, while all the rest of the ports are mapped to the second buffer.
Results from test 1
Ports 9-24 are mapped to the same buffer, and as a result all access the same shared resource, which means they have the same bandwidth.
Results from test 5
Port 8 receives 100% of one buffer, which translated to 50% of the port bandwidth!
All other ports (15 ports) when grouped together use 100% of the second buffer, which translates into 50% of the port bandwidth (for all the 15 ports). Results are also illustrated in the pie charts that follow:
How Can I Reproduce the Tolly Unfairness Results?
As the Tomahawk can be wired differently on each switch system, it depends on the board's design and on the number of ports you have available on the test equipment.
Basically you need to find the mapping between the two buffer pools:
In this case:
- Group 1: Ports 1-8, 25-32
- Group 2: Ports 9-24
Once you understand this mapping, you can take one port from group 1, and the rest of the ports in group 2, and generate traffic to an egress port (any port).
For example as in our case in test 5 (which shows the most unbalanced load):
- Group 1 used port 8.
- Group 2 used the remaining ports (9-23).
When the mapping is different from this example, you need to perform some tests to identify the mapping of the ports to groups.
Note: If you randomly take 5 ports while 4 ports are sending traffic to one port, there is a great chance that you will see unfairness in the results, as not all the ports will likely to be in the same buffer pool.
What Testing Tool Can I Use?
Any testing tool can be used to show when unfairness is encountered. If IXIA/Spirent is available, use it. Otherwise you can use any Linux traffic generator. For example, use the perftest package (ib_send_bw), iperf, or any other tool that can reach a line rate that approximates these conditions.
What Contributes to Unfairness?
Let's look on the shared buffer and use test 5 as an example.
In this test we injected 100Gb/s traffic from each port with Port 8 mapped to one pool (in this case Pool 0) of the shared buffer that is equal to 50% of the size of the buffer. When there is congestion, all of the pool will be full. Ports 9-23 are mapped to the second pool (in this case Pool 1) of the shared buffer, which is also equal to 50% of the full size of the buffer. When it encounters congestion, this pool will be full as well.
When Pool 0 is full, Port 8 will have 100% chance to get in the buffers, assuming there is a place for one packet in the buffer.
When Pool 1 is full, each port out of Ports 9-23 (15 ports) will have 1/15th chance to access the buffers, assuming there is a place for one packet.
The egress scheduler selects the packets according to the time arrived (FIFO).
Let's look at the egress queue:
Port 8 gets 50% of the bandwidth, while all the remaining the ports can get only 50%.
Can This Problem Be Solved in Software for Tomahawk IC?
No, the conditions highlighted in the Tolly Report are characteristic of the Tomahawk's IC design.
Why Don't We See This Behavior in Spectrum IC?
Spectrum IC has fully shared buffer.