Abstract:Deep neural networks with applications from computer vision to medical diagnosis 1,2,3,4,5 are commonly implemented using clock-based processors 6,7,8,9,10,11,12,13,14 , in which computation speed is mainly limited by the clock frequency and the memory access time. In the optical domain, despite advances in photonic computation 15,16,17 , the lack of scalable on-chip optical non-linearity and the loss of photonic devices limit the scalability of optical deep networks. Here we report an integrated end-to-end photonic deep neural network (PDNN) that performs sub-nanosecond image classification through direct processing of the optical waves impinging on the on-chip pixel array as they propagate through layers of neurons. In each neuron, linear computation is performed optically and the non-linear activation function is realized opto-electronically, allowing a classification time of under 570 ps, which is comparable with a single clock cycle of state-of-the-art digital platforms. A uniformly distributed supply light provides the same per-neuron optical output range, allowing scalability to large-scale PDNNs. Two-class and four-class classification of handwritten letters with accuracies higher than 93.8% and 89.8%, respectively, is demonstrated. Direct, clock-less processing of optical data eliminates analogue-to-digital conversion and the requirement for a large memory module, allowing faster and more energy efficient neural networks for the next generations of deep learning systems.

Photon-limited non-imaging object detection and classification based on single-pixel imaging system

Noise-resilient single-pixel compressive sensing with single photon counting

Robust and efficient single-pixel image classification with nonlinear optics

Single-pixel Multispectral Imaging Based on Macropixel Segmentation Method

Lensless compressive sensing with annulus-sector-shaped pixel geometry in the photon-starved environment

High precision single-photon object detection via deep neural networks

Image Classification in the Dark using Quanta Image Sensors

Non-imaging single-pixel sensing with optimized binary modulation

A Super-resolution Optical Classifier with High Photon Efficiency

Image-free real-time classification of fast moving objects using 'learned' spatial light modulation and a single-pixel detector

Single-pixel imaging based on deep learning

Photon-Efficient 3D Imaging with A Non-local Neural Network

Multiframe-integrated, In-Sensor Computing Using Persistent Photoconductivity

A simple and robust single-pixel computational ghost imaging

Single-Pixel Photon-Counting Imaging Based on Dual-Comb Interferometry

Classification and reconstruction for single-pixel imaging with classical and quantum neural networks

An on-chip photonic deep neural network for image classification

Robust photon-efficient imaging using a pixel-wise residual shrinkage network

Accurate Unsupervised Photon Counting from Transition Edge Sensor Signals

Photon-Efficient Computational 3D and Reflectivity Imaging with Single-Photon Detectors

Deep-learning-based Single-Photon-counting Compressive Imaging Via Jointly Trained Subpixel Convolution Sampling.