Introduction to Spectrum, the 100GbE Switch Silicon

Version 21

    This post discusses the feature list, capabilities and various solution examples of the next generation 100GbE switching IC named Spectrum. In addition, we will touch the new SN2000 switch system family based on this IC.
    This post is basic, and meant for IT managers, architects and technical people.

     

    References

     

    Spectrum Overview

    Spectrum is the next generation of switching IC, delivering leading throughput, low-latency and scalability for Ethernet networks. Spectrum is equipped with 32 non-blocking 100GbE ports, 32 ports of 40GbE or up to 64 ports of 10/25/50 GbE, all operate with zero packet loss. Spectrum was designed for various markets such as storage, cloud, data centers, hyperscale networks and more. The Spectrum IC excels in its low power consumption of around 4W per 100GbE port, or 135W for the entire IC, running in maximum speed and throughput.

     

     

    600px_spectrum_no_logo.png

     

    Features

    Refer to the Product Briefs of the Spectrum switch IC and the switch systems (SN2100, SN2410 and SN2700)

     

    SN2000 Series 100GbE Switch Systems

    The next generation switch series based on Spectrum is the SN2000 series, of which the first introduced switch is the SN2700.

     

    Switch ModelPort Configuration
    SN2100

    16x40/100GbE

    Port Split options: 64x10/25GbE, 32x50GbE

    SN2410

    48 SFP+ ports of 10/25GbE and 8 ports of 100GbE

    SN2700

    32x100GbE

    Port splitting option - Up to 64 ports of 10/25/40/50 GbE using split/reduced cables

    SN2700: 100GbE Switch System

     

     

    SN2700_small.png

     

    Solution Highlights

     

    Performance

    Non-blocking, deterministic 3.2Tb/s switching capacity

    Spectrum provides a non-blocking, deterministic 3.2Tb/s switching capacity. It delivers full wire speed at any packet size for L2/L3/MPLS, and would excel in any benchmark, such as RFC 2544 tests.

    This level of performance is unmatchable, enabling a network to be free from packet loss.

     

    Predictable low-latency

    With a cut-through latency under 300ns, Spectrum delivers consistent application performance and fast access to new generations of storage devices such as flash memory. Spectrum's cut-through latency is not dependent on the load, so even with heavy loads and full capacity, low latency is being kept. For any mix of ports speeds at the device, such as Top-of-Rack configurations, where the uplink/aggregation ports are usually configured to a faster rate than the downlink/access ports, Spectrum maintains the consistent cut-through latency between the access ports and from the aggregation ports towards the access. To learn more about Store and Forward vs. Cut-though click here.

     

     

    cut-through.PNG

     

    Lowest Power

    Spectrum delivers the lowest power consumption, enabling infrastructure managers to reduce operations expenses (OPEX) with around 4W per 100GbE port, for a total of 135W maximum power. Not only reducing OPEX, this energy efficient operation allows more efficient switch products in terms of building simpler switch systems, reducing the need for cooling etc.

     

    Scalable Cloud and Virtualized Networks

    L2 overlay networks - VXLAN, GENEVE, NVGRE

    With flexible and scalable architecture, Spectrum enables cloud and data-center operators to scale their support for virtual networks. With native support for routing the overlay traffic, prior to processing of the underlay traffic, Spectrum enables building optimal environment for hosting tenants with multiple subnets without the need for remote, low performance and expensive router. Its scalable and shared databases allow the tuning of databases according to needs, either for an dense-overlay-MAC-address network which supports many VMs behind each VTEP, or sparse across many VTEPs, which consumes many underlay IP addresses.

     

    vtep.PNG

     

     

    L3 overlay networks and tunnels - IPinIP and GRE tunnels

    Tens of thousandths of tunnels are supported by Spectrum eliminating the constrains for the heavily virtualized network, enables connecting anyone to anywhere by tunnels.

     

    L3 overlay networks and tunnels - MPLS

    Tens of thousandths of MPLS tunnels are supported by Spectrum, eliminating the constrains for the MPLS deployments. With emerging use of MPLS in the data centers and their borders, Spectrum architecture supports ultra-scale of MPLS tunnels, with premium support for new standards such as SPRING, HSDN, ECMP, entropy labels and more. Network architects do not need to choose between best-of-class utilization of the network using ECMP and scalable MPLS implementations, they can get them both at the same time in a single solution.

     

    Advance Monitoring Features

    Spectrum delivers best in class monitoring tools that enables the cloud and data-center operators to detect, control and debug their network.

     

    Virtual networks visibility

    Spectrum enables simple and scalable visibility into the virtual networks' traffic that traverse within the cloud and the data center. These capabilities are enabled thanks to the ability to match the overlay headers of the packets with simple rules, and apply counters, policers and mirroring agents accordingly. By that, the network manager can understand the routes taken by a tenant or a customer, control that only allowed traffic reaches restricted environments of the network and so on.

     

    Congestion and micro-burst monitoring

    Spectrum implements multiple mechanisms to allow the network operators to get the most relevant and fresh information regarding the networks status in terms of congestion and micro-bursts. Modern clouds and data-centers usually face congestion issues due to the high demand for bandwidth, either for north-south or the fast-growing east-west traffic. An operator that is aware of the potential network congestion can understand the performance implications that may affect its customers and react to prevent these implications.

    The tools that are supported are:

    • Per queue buffer status
    • Per queue watermarks to detect the longest queue status detected
    • Hardware based histograms to detect the fine-grained behavior of the queue over time
    • Transmission rate status and hardware based histograms
    • Congestion-triggered mirroring
    • Buffers and WRED drop counters
    • Per hash bucket counter to enable load balancing control

     

    Load balancing (ECMP and LAG)

    Spectrum enables fine-grained, consistent and controlled ECMP and LAG. The Network manager can monitor the load balancing mechanism using counters per hash-buckets, and re-balance the load among the interfaces according to the counters or changes that happen in an ECMP or LAG group. By that, the network manager prevents uneven distribution of traffic and rehashing of the traffic. A typical example is the case of a port link down of a group member.

    Load balancing in a virtual network

    As cloud and data center operators strive to utilize their network, one of their most important tool is to load balance the traffic among different interfaces. The problem that may arise is the amount of flows that can be detected by looking at the underlay headers only, whatever protocol is being used. In order to help spreading the traffic, Spectrum detects and defines the flows using both the underlay and the overlay headers, without the need to choose only a small amount of fields. By that, Spectrum allows fine-grain load balancing for the underlay network manager for achieving the best utilization of the data-center's infrastructure.

    Operators that choose to virtualize their networks or building tunnels using MPLS may face the above hurdles as well, and even to a higher degree, since the large amount of labels that are not always pre-configured. Spectrum overcomes these hurdles by allowing a simple configuration of the fields of the overlay headers and applying them into the ECMP and LAG mechanism, regardless the amount of labels that the packet is carried with.