Abstract:With the proliferation of ultrahigh-speed mobile networks and internet-connected devices, along with the rise of artificial intelligence (AI)<a href="/articles/s41586-020-03070-1#ref-CR1">1</a>, the world is generating exponentially increasing amounts of data that need to be processed in a fast and efficient way. Highly parallelized, fast and scalable hardware is therefore becoming progressively more important<a href="/articles/s41586-020-03070-1#ref-CR2">2</a>. Here we demonstrate a computationally specific integrated photonic hardware accelerator (tensor core) that is capable of operating at speeds of trillions of multiply-accumulate operations per second (1012 MAC operations per second or tera-MACs per second). The tensor core can be considered as the optical analogue of an application-specific integrated circuit (ASIC). It achieves parallelized photonic in-memory computing using phase-change-material memory arrays and photonic chip-based optical frequency combs (soliton microcombs<a href="/articles/s41586-020-03070-1#ref-CR3">3</a>). The computation is reduced to measuring the optical transmission of reconfigurable and non-resonant passive components and can operate at a bandwidth exceeding 14 gigahertz, limited only by the speed of the modulators and photodetectors. Given recent advances in hybrid integration of soliton microcombs at microwave line rates<a href="#ref-CR3">3</a>,<a href="#ref-CR4">4</a>,<a href="/articles/s41586-020-03070-1#ref-CR5">5</a>, ultralow-loss silicon nitride waveguides<a href="/articles/s41586-020-03070-1#ref-CR6">6</a>,<a href="/articles/s41586-020-03070-1#ref-CR7">7</a>, and high-speed on-chip detectors and modulators, our approach provides a path towards full complementary metal–oxide–semiconductor (CMOS) wafer-scale integration of the photonic tensor core. Although we focus on convolutional processing, more generally our results indicate the potential of integrated photonics for parallel, fast, and efficient computational hardware in data-heavy AI applications such as autonomous driving, live video processing, and next-generation cloud computing services.

Integrated Photonic Encoder for Terapixel Image Processing

Integrated photonic encoder for low power and high-speed image processing

A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture

Compressive Single-Pixel Hyperspectral Imaging Using RGB Sensors

An Encoded Photodetector Array for High-frame-rate Compressed Sensing and Computational Imaging

A Parallel Photonic Chip for Nano-Second End-to-end Image Processing, Transmission, and Reconstruction

Highly Efficient Single-Pixel Imaging System Based on the STEAM Structure

Photonics-enabled Compressive Sensing with Spectral Encoding Using an Incoherent Broadband Source.

Photonic real time video image signal processor at 17Tb/s based on a Kerr microcomb

High-Speed Real-Time Image Compression Based on All-Optical Discrete Cosine Transformation

Hypermultiplexed Integrated-Photonics-based Tensor Optical Processor

Development and Application of an Integrated Laser-Enabled Silicon Photonic Tensor Core

PIXEL: Photonic Neural Network Accelerator

Large-Scale Optical Neural Networks based on Photoelectric Multiplication

Parallel convolutional processing using an integrated photonic tensor core

Integrated electro-optic digital-to-analog link for efficient computing and arbitrary waveform generation

Terahertz Deep-Optics Imaging Enabled by Perfect Lens-Initialized Optical and Electronic Neural Networks

All-analog photoelectronic chip for high-speed vision tasks

Seven Bit Nonvolatile Electrically Programmable Photonics Based on Phase-Change Materials for Image Recognition

Hypermultiplexed Integrated Tensor Optical Processor

6.9 A 0.35V 0.367TOPS/W Image Sensor with 3-Layer Optical-Electronic Hybrid Convolutional Neural Network