Active Optical Cables (AOC) are widely used in HPCs and have more recently became popular in hyperscale, enterprise and storage systems as a high-speed, plug & play solution with longer reaches than Direct Attach Copper (DAC) cables.
What is an AOC?
Optical transceivers convert electrical data signals into blinking laser light which is then transmitted over an optical fiber. Optical transceivers have a detachable optical connector to disconnect the fiber from the transceiver. AOCs bond the fiber connection inside the transceiver end, creating a complete cable assembly much like a DAC cable, only with a 3-200-meter reach capability. AOCs main benefit is the very long reach of optical technology, while acting like a simple, “plug & play” copper cable. A complete 2 transceiver AOC including fiber cable assembly is usually priced near the price of only a single connectorized transceiver.
AOC Features and Advantages
Compared to less expensive DAC cables, AOCs offer:
- Longer reach capability than DAC 3-7 meter limits
- 3-100 meter multi-mode technology
- 100-200 meters with Mellanox single-mode, Silicon Photonics technologies
- Thinner, lighter, and tighter bend radius cables enabling more flexible configurations, increased airflow for cooling, and easier system maintenance
Compared to more expensive optical transceivers, AOCs offer:
- Dramatically lower priced solution than two optical transceivers and connectorized fiber based links
- Lower power consumption at 2.2 Watts versus up to 2.6 to 4.5 Watts for optical transceivers (4-channel)
- Lower operational and maintenance cost
The following photo shows the use of thousands of AOCs in a HPC supercomputer at the University of Texas. The AOCs are precisely manufactured with all eight fibers cut to exact lengths to minimize the time skew between each of the four channels within the AOC cable end. This is to enable each of the four individual signal pulses to arrive at the transceiver end at exactly the same time. Believe it or not, the speed of light delay traveling at 186,000 miles per second in the fiber is a significant issue over a 10-meter AOC in an HPC supercomputer. High-speed computing is all about minimizing latency delays between critical components.
How AOCs Differ from Other Interconnect Solutions
Permanently attaching the fibers is a seemingly simple change but yields a surprisingly large number of technical benefits and cost advantages; enough to create an entirely new category of interconnect products. Since the optics are contained inside the cable, designers do not have to comply to IEEE or IBTA industry standards for transceiver interoperability with other vendors. This gives designers complete freedom to pick and choose the lowest cost, best performing technologies since the cable is a closed system and has a predefined cable length. All of this results in dramatic cost and price reductions. Here are some of the results this simple change enables:
- Lowest priced optical interconnect available: Near half the price of a single optical transceiver – much more than just the cost of deleting the optical connectors
- Plug and play: Ease-of-use cable features – like DAC cables only with longer optical reach
- Long reach: Up to 100 and 200-meter reach depending on the technology
- Lowest optical power consumption per end: Significantly lower than connectorized transceivers – saves operating expenses in power consumption and cooling
- No optical connectors to clean and maintain: Saves operating expenses and increases reliability
- Optical isolation: Isolates electrical systems from ground loops as with copper DAC cables – a reliability advantage
Achieving Lower Product Cost
An AOC uses two optical transceivers with integrated fibers. So, how does it cost on par with a single optical transceiver with an optical connector?
- Testing Costs: Optical testing accounts for 40-50 percent of the total cost of manufacturing a transceiver. AOCs can be tested in a switch system as an electrical test. You plug the cable in; test patterns and data run; come back later and look at the results. If all is good, ship! If not, scrap! Optical transceivers, on the other hand, are much more complex, requiring $500,000 worth of optical test equipment per station, a very experienced (e.g. expensive) test technician, and a lot of manual time on the test bench. AOCs do away with all of this since the testing is only in the electrical domain. Mellanox uses its “scratch & dent” switches to test AOCs and is one way we achieve a bit error ratio (BER) of 1E-15 versus 1E-12 IEEE standard, about 1,000 times fewer number of bit errors induced by the AOC link.
- Design Freedom: Since the optics are contained inside the AOC cable, designers can utilize the lowest cost materials and transceiver designs. Besides deleting four MPO optical connectors (2 per end), for example, Mellanox’s silicon photonics AOC uses only one laser versus four for a single-mode AOC. VCSELs that do not qualify for transceivers can be reused and low cost, short reach (orange colored) OM2 fiber can be used for <20 meter reaches saving more expensive OM3 and OM4 fiber for longer reaches.
- Freedom from Industry Standards: AOCs must comply to the IEEE, IBTA and SFF industry standards for the electrical, mechanical, and thermal requirements but the hardest part to comply with are the optical requirements. Since the optics are contained inside the cable, they do not have to meet any standards allowing for a lot more design freedom and material use and eliminating the need for costly optical testing.
AOCs Offer Lower Operational “Hidden Costs” Too
- AOCs do not have optical connectors that require manual cleaning every time they are removed as a single speck of dust inside the connector can completely block the 50-um or 9-um diameter fiber light transmission area. In a transceiver link, there are two fiber ends and two transceiver ends to clean. Besides, the personnel cost the connector cleaners can cost upwards of $250 each.
- AOCs don’t use MPO or LC optical connectors which, in crowded racks, may be dropped, scratching the fiber-end and rendering them useless.
- Optical connectors can channel an electrical static charge that builds up on a long plastic cable and can destroy the sensitive optical transceiver electronics when connected.
- AOC is a “plug and play” cable solution rather than a “plug, assemble, and clean” solution as with optical transceivers. Optical transceivers, fibers, and connectors also have many different and complicated product variances. All these must all be exactly matched to the specific transceiver used and spares would need to be kept handy as well as a technician trained in the specifications.
- AOC cables have a short bend radius and are much thinner than most DAC cables. This makes them easier to deploy and frees up a lot of space for increased airflow and cooling in crowded systems.
- MPO optical connectors are known to insert half way into the transceiver and look fine to the technician later creating problems and maintenance issues.
- Lastly, there are big operational savings in power consumption costs. One Watt saved at the component level translates to 3-5 Watts at the data center facility level. This is when all the chassis, row, room and facility fans and air conditioning equipment is included along with the electrical power to drive them – not counting the repair and maintenance! AOCs are less complex than optical transceivers and offer lower power consumption. Mellanox designs its own AOC ICs, so can offer incredibly low power consumption ratings of 2.2 Watts per end.
AOC Use in Modern Data Centers
While AOC reaches can extend to the limits of the optical technology used (100-200 meters), installing a long 100-meter (328 foot) cable, complete with an expensive transceiver end, is difficult in crowded data center racks so the average reach typically used is between 3-30 meters. Only one “oops” per cable allowed. Damaging the cable means replacing it as it cannot be repaired in the field. AOCs are typically deployed in open access areas such as within racks or in open cable trays for this reason.
Mellanox’s InfiniBand AOCs started out in about 2005 with DDR (4x5Gb/s) for use in the Top10 HPCs and quickly became the preferred solution for the large InfiniBand HPCs in the Top100 in which Mellanox is the market leader. Today, AOC use is the norm for QDR, FDR and EDR and in 2017 the newly announced LinkX 200Gb/s HDR announced at SC’16 event in November.
The power and cost savings caught the eye of the Ethernet hyperscale and enterprise data center builders and has since become a popular way to link Top-of-Rack switches upwards to aggregation layer switches such as End-of-Row and leaf switches. Several hyperscale companies have publicly stated their preferred use of AOCs for linking Top-of-Rack switches. Additionally, single channel (SFP) AOCs have become very popular in high-speed, NVMe storage subsystems. Some hyperscale builders often run 10Gb/s or 25Gb/s AOCs from a Top-of-Rack switch to subsystems at reaches greater than DAC limits of 3-7 meters.
On the down side, AOCs have an expensive transceiver end which is difficult to install in crowded areas over long reaches, hence the average use is < 20-30 meters. AOCs are typically deployed where there is easy access to cable trays or open areas. When an AOC fails, most operators simply abandon it in place and run another AOC.
Here is an example of how AOCs are typically used inside systems racks to link subsystems together, between switches, and across systems in rows:
Here is a more detailed view of Ethernet configurations showing Mellanox’s LinkX 10Gb/s and 25Gb/s based AOCs, Spectrum switches and ConnectX-3, ConnectX-4, and ConnectX-5 QSFP and SFP network adapters. Additionally, Mellanox recently announced two new AOCs that break out to 25Gb/s and 50Gb/s ends. A 100Gb/s QSFP28 AOC split out into four 25Gb/s SFP28s ends and another cable with a dual 50Gb/s QSFP28 break-out.
Why Choose Mellanox AOCs?
Mellanox not only designs its own AOC control ICs such as a TIA/Laser Driver but the company designs, manufactures and tests every AOC cable as well. The vertical integration of components enables maximizing the signal margin at every point.
Mellanox was one of the first adopters of AOCs in high-speed HPC supercomputers and has a long history and experience in AOC deployment and manufacturing.