Long Reach Optics in the Data Center

Version 4

    This post discusses the usage of Long Reach (LR) single-mode transceivers in the data center.

     

     

    References

     

    What is Single-mode Optics?

    Single-mode optics is all about long reaches. It is enabled by using single-mode fiber – the same fiber used by the telecom industry to send Internet data between continents and across oceans. Traditionally, single-mode transceivers have been exceptionally expensive but Hyperscale builders began ordering these in huge volumes are driving down the cost.

     

    Single-mode transceivers use various technologies to achieve price and reach targets. Lasers are made using Indium Phosphide (InP) and emit in the 1310nm and 155nm wavelengths. Lasers can be directly modulated via turning on and off electrical currents (called DMLs) or use continuous wave lasers and modulating the light externally (called EAs or EMLs). Lasers, modulators, waveguides, multiplexers, detectors, and other elements all have to be integrated or precision aligned to “bacteria level” tolerances of hundreds of nanometers. It is difficult to do at all, much less to make a million reliable attractively priced units.

     

    Silicon Photonics – Solves Costly Alignment Manufacturing Problems

    Since silicon is transparent to both 1310 and 1550nm wavelengths, silicon wafers can be built which channel light into waveguides inside the wafer surface to various optical elements embedded in the material system in the wafer. Lasers are built using InP and attached to the silicon photonics chip and the chip is then attached to optical fibers. Called silicon photonics, it is primarily a manufacturing technique which uses semiconductor processes to precisely align optical elements thereby dramatically reducing manufacturing costs for single-mode transceivers which can account for upwards of 50% of the manufacturing total cost.

     

    Mellanox is a leading supplier of silicon photonics transceiver technologies in both transceivers and silicon photonics engine components. Mellanox currently offers 1550nm-based PSM4 transceivers in QSFP28 package with a 2km reach capability. Mellanox builds PSM4s using its internally developed silicon photonics technologies located in southern California which has been building and shipping silicon photonics transceivers and Variable Optical Attenuators (VOAs) products for over a decade.

     

     

    Single-mode Transceiver Advantages

    • Enables using low-cost, long fiber reaches up to 10km
    • Makes the type of fiber and reach used anywhere in the data center a non-issue
    • Can support hundreds of wavelengths in a single fiber
    • Largely data rate speed agnostic
    • 1310/1550nm wavelengths enable using silicon photonics manufacturing technologies

     

    The main advantage of single-mode transceivers is that it enables using inexpensive single-mode fiber that has a very long reach capability. Recently, as transceiver costs drop, it is also being used for in-rack breakout applications like DAC, AOCs, and multi-mode transceiver configurations mentioned above. The long reach capability makes it one of the most flexible interconnect types as it makes reaches in a data center a “non-issue”. By employing a tiny 9um core optical fiber, the data signal pulse is recognizable at the receiver over an astounding 100km reach! Which is why the telecom industry uses single-mode fiber to connect cities and countries together.

     

    Large data centers are moving to deploying single-mode fiber as fast as they can to get rid of the numerous multimode fiber short reach hops and fiber/connector complexities required to send data past 100m. These are costly, expensive to maintain, and must be constantly upgraded with each line rate speed advances. With speed advances happening every 2-3 years instead of 5-8, data center operators are looking to eliminate these costs. Multi-mode fiber is tuned to specific data rates and bandwidths. Every increase in data rates means either the multi-mode fiber reach becomes shorter, or the fiber needs to be upgraded (OM2, OM3, OM4 now OM5). Add to this the complexity of many different types of MPO connectors. Large data centers have very complex networks inside that create optical losses as well as having enormous physical data centers (as long as 1km) and they also need to connect to the metro area network using 10km transceivers. Costs and long reach is what is driving the popularity of single-mode optics in data centers.

     

    To go 200m, four SR4 transceivers would be needed with a network switch in between. With single-mode optics, 2 transceivers alone are capable of sending data 10km! While single-mode transceivers are more expensive than multimode transceivers, when considering a long link, the total cost of fiber, connectors, and transceivers is less with single-mode optics.

     

    Hyperscale data centers want to build a huge data center infrastructure once (some costing up to $1 billion) and connect everything with inexpensive single-mode fiber from the core router across the data center all the way to each individual server inside racks and then to never touch it again instead of, as with multimode fiber, to change the infrastructure every few years. Transceiver pricing is holding them back but silicon photonics may provide a solution.

     

    What Data Centers are 10KM Long?

    While most data centers are not 2km or 10km long, the “km” spec is another way of stating the optical power of the laser. Measured in dBs, the Mellanox PSM4 offers ~3.3 dBs of optical power (LR4 has an optical power of 6dBs) which is enough to push through hundreds of meters of a lossy fiber infrastructure consisting of dirty and/or misaligned optical connectors, jumpers, optical patch panels and other interferences to the light path. This is like needing a very powerful flashlight to shine through a dense forest of twigs, branches and leaves in the way even though the actual distance is relatively short.

     

    Multiplexing

    Hundreds of different colored laser pulses can be simultaneously sent over a strand of single-mode fiber. Telecom does 1,000 but in data centers today four wavelengths is common (soon headed for 8).  Additionally, single-mode fiber is data rate agnostic, unlike multi-mode fiber. Unlike multi-mode fiber, the same fiber can be used for 1Gb/s, 10Gb/s, 25Gb/s, 50Gb/s, or 100Gb/s individual line rates with little impact on the reach. Using silicon photonics to reduce manufacturing costs and sending multiple 100Gb/s data streams with different colored laser down a single strand of low-cost single-mode fiber is the technology goal of the future.

     

    Single-mode Transceivers in Modern Data Centers

    The main single-mode transceiver types used in data centers today are:

    • LR:             10Gb/s, 25Gb/s, SFP, 1-channel, 2-fiber, LC connector, 10km, 1310nm
    • PSM4         40Gb/s, 100Gb/s QSFP, 4-channel, 8-fiber, MPO/APC connector, 500m-2km, 1550/1310nm
    • CWDM4   40Gb/s, 100Gb/s QSFP, 4-channel, 2-fiber, LC connector, 2km, 1310nm
    • LR4            40Gb/s, 100Gb/s QSFP, 4-channel, 2-fiber, LC connector, 10km, 1310nm

     

    PSM4 sends each channel into separate parallel fibers–one per channel. CWDM4 and LR4 multiplex 4-channels each with different wavelength lasers into a single fiber and de-multiplexes them at the receiver end–sending a “rainbow” down the fiber. PSM4s are generally used at less than 500m, CWDM4 up to 2km, and LR4 to 10km.

     

     

     

     

     

    Numerous PSM4 Applications and Configurations for Any Need

    The PSM4 has many different configuration application uses and is one of the hottest selling transceivers in the hyperscale segment. It can bus 100Gb/s point-to-point over 2km or can be broken out using passive fiber splitter cables or half-AOC hybrid (called a, “pigtail”) into dual 50Gb/s or quad 25Gb/s links for linking to servers, storage and other subsystems within a rack.

     

    1550-1310nm PSM4 Interoperability

    While Mellanox’s PSM4 uses the 1550nm wavelength, most PSM4 transceivers use a wide bandwidth detector so transceivers with either wavelength can interoperate with ease.

     

    PSM4 Breakouts to Servers & Storage

    Beside long reach 2km point-to-point links, PSM4 channels can also be split out individually. The diagram below shows a 100Gb/s PSM4 transceiver split using a passive breakout splitter cable with an MPO on one end and either dual MPOs (50Gb/s) or quad LC connectors (25Gb/s) on the other end. CWDM4s and LR4s cannot do this feature and can only bus 100Gb/s point-to-point.

     

    Passive Fiber Breakout Configurations

     

     

    This figure shows all the different uses of single-mode optics in modern data centers from core routers as far away as 10km all the way to individual servers. Using various single-mode transceivers and fiber combinations.

     

     

    Why Choose Mellanox Single-mode Transceivers?

    Mellanox designs its own PSM4 TIA and Laser Driver transceiver control ICs. In addition, the company has over 20 years of experience in designing and building silicon photonics transceivers and VOAs in an in-house silicon photonics fab and outside 12-inch fab.

     

    Mellanox designs, manufactures, and tests every transceiver in live switching systems unlike competitors who test with benchtop equipment. BER ratings are 1E-15 and 1,000 better than IEEE industry standards and competitors at 1E-12.

     

    The vertical integration of components combined with manufacturing all from one company enables maximizing the signal margin at every point, power consumption, and transceiver reliability. Mellanox’s 100Gb/s PSM4 has a reach of 2km versus the PSM4.MSA specification at 500m. Mellanox also offers the PSM4 transceiver components as individual parts so OEMs can integrate 100Gb/s PSM4 transceivers into their designs at the chip level.

     

    Key single-mode transceiver features:

    • Enables long reaches up to 10km making various reaches a non-issue
    • Employs inexpensive single-mode fiber
    • Line rate agnostic supporting 100Gb/s per channel futures
    • 1310/1550nm wavelength enables employing silicon photonics to reduce manufacturing costs
    • Enables multiplexing multiple channels into a single fiber