Short Reach Optics in Data Centers

Version 4

    This post makes an introduction to Short Reach (SR) transceiver usage in data centers.

     

    References

     

    Overview

    Short Reach (SR) multimode optics are the lowest priced optical interconnects available today that use detachable optical connectors to separate the transceiver from the optical fibers. Although AOCs and SR transceivers both support 100m reaches, AOCs are much less expensive to manufacture but due to installation difficulties, are generally used at less than 20-30 meter reaches. However, SR optics are frequently used up to 80-100 meter reaches. The key advantage of SR transceivers is that the transceiver can be separated from the optical fiber infrastructure which may be permanently installed in structured cabling pipes, under computer room floors, or even between multiple floors.

     

    Short reach optics are not new and have a long history of different fibers, connectors, and transceiver types at different data rates creating a blizzard of different parts and complex, specific features to each. But modern data centers have simplified things and zeroed in on the single-channel SFP with the duplex 2-fiber LC optical connector and the four-channel QSFP with the 8-fiber MPO connector. Mellanox offers both of these configurations as 10Gb/s and 25Gb/s line rates and quad versions at 40Gb/s and 100Gb/s.

     

    Multimode fibers have a large 50um diameter light carrying fiber core. This makes SR transceivers easier and less expensive to manufacture compared to single-mode optics with tiny 9um fiber cores which are difficult and relatively expensive to build with. For purposes of lowering costs, multimode, short-reach optics are very popular in modern hyperscale, enterprise, and storage data center applications as an optical solution. Approximately 70-80% of the links in a data center are at reaches less than 60 meters traveling up and down rows of racks, and well within the 100-meter reach of short reach transceivers.

     

     

    VCSEL Multimode Lasers & Fibers

    Short reach transceivers use a laser that is built on a gallium arsenide (GaAs) semiconductor wafer and constructed perpendicular to the surface of the wafer. When excited by electrons, GaAs emits light at 850nm wavelength and is channeled into a vertical cavity on the wafer surface where it resonates and becomes laser light. Hence the name, Vertical Cavity, Surface Emitting Laser or VCSEL.

    Multimode optics employ a large core diameter, 50um, optical fiber that is easy to interface to VCSEL lasers and detectors, so the costs are much lower than single-mode optics with a tiny 9um core diameter fiber which are difficult to align. But the SR laser pulse tends to scatter into multiple transmission paths or “modes” (hence the name multimode). The scattered pulse in large diameter fibers becomes unrecognizable after about 100 meters, so the IEEE standards body sets the limit at 100 meters, assuming four connectors are in the run. Multimode can reach up to 400m but requires specialized lasers, fibers, and connectors which are priced near that of single-mode transceivers. For reaches longer than 100m, single-mode optics are generally used.

     

     

    Connectorized Optics A Key Feature

    Many data centers have structured cabling where the fiber infrastructure is fixed and installed in cabling pipes under raised floors, and integrated into optical patch panels used to manually reconfigure the fiber run end points. Sometimes, fibers run to other system rows, rooms, floors, or even other buildings necessitating the ability to disconnect the fibers from the transceivers installed in the systems. This is something that DAC and AOCs integrated cables cannot do as the wires or fibers are integrated into the plug or transceiver end. Multimode optics use the 2-fiber LC and the 8-fiber MPO optical connectors.

     

    Point-to-Point Applications: ToR-to-Leaf/Spine EOR Switches

    One of the main applications for SR and SR4 transceivers are to link Top-of-Rack (ToR) switches to other parts of the network such as aggregation switches, middle, and end-of-row switches, and to leafs in a leaf-spine network. These are typically used as high bandwidth buses that are four-channel SR4s at 40G or 100Gb/s bandwidths. Single channel SR 10G and 25G transceivers can be used to link server and storage systems to a ToR switch within a rack or adjacent racks. Since the number of links is very high the closer to the server one gets, low cost optics is important and multi-mode optics is well suited to these applications where the reaches are relatively short spanning within a rack or along a single row.

    While several enormous hyperscale operators have made a lot of noise in the press around moving to single-mode fiber, many big hyperscale and enterprise installation still operate as groups of small system clusters where all the systems are well within the reach of 100m multi-mode fiber. Interestingly multi-mode fiber is about three times more expensive than single-mode fiber, the single-mode transceivers are 50 percent to 10X more expensive than multi-mode transceivers. Single-mode transceivers are difficult to build but offer reaches up to 10Km vs only 100m of multi-mode. Single-mode fiber is the mainstay of the telecom industry linking cities and countries together hence is made in thousand of mile spools.

     

    Breakout Applications: ToR QSFP Breakouts to SFP Servers & Storage

    Linking Top-of-Rack switches down to servers and storage subsystems within the same rack is another popular use for SR and SR4 optics. In the past, SR4 transceivers only transfer at 4-channels at a time to another SR4 switch-to-switch. Newer transceiver models can split the four into individual single-channels that can be connected to different systems and operate independently. This is important when the link reach needed is greater than the 3-meter capability of DAC copper cables and perhaps spanning more than one rack. The passive fiber break-out cable has a single 4-channel MPO on one end connecting to the SR4 transceiver and four Duplex LC optical connectors on the other end connecting to four separate SFP transceivers each with their own 100m fiber run.

     

    Similarly, two 50Gb/s links can be created from one 100Gb/s using an MPO breakout cable with two MPOs connected to 50Gb/s SR2 transceivers using only two channels each (2x25G) as shown in the figure below (100G QSFP28 Breakouts to Dual 50G QSFP28 and Quad 25G SFP28s Breakouts Using 8-fiber MPO and 2-fiber LC Optical Connectors).