Abstract:Convolutional neural networks (CNNs) are representative models of artificial neural networks (ANNs). However, the considerable power consumption and limited computing speed of electrical computing platforms restrict further CNN development on edge devices. Optical neural networks are considered next-generation physical implementations of ANNs, but their capabilities are limited by on-chip integration scale and requirement for coherent light sources. This study proposes a spectral convolutional neural network (SCNN) of incoherent natural light by an optical convolutional layer (OCL) and a reconfigurable electrical backend. The OCL is implemented by integrating very large-scale, pixel-aligned spectral filters on a CMOS image sensor on a 12-inch wafer, facilitating highly parallel spectral vector-inner products of incident light. It accepts broadband incoherent natural light containing two spatial and one spectral dimension directly as input with the function of matter meta-imaging. This unique optoelectronic framework empowers in-sensor optical analog computing at extremely high energy efficiency because the OCL is driven by the energy of the information carrier, i. e. natural light. To the best of our knowledge, this is the first integrated optical computing utilizing natural light. We employ the same SCNN chip for completely different real-world complex tasks,and achieve accuracies of over 96% for pathological diagnosis and almost 100% for face anti-spoofing at video rates. The SCNN framework has an unprecedented new function of substance identification, provides a feasible optoelectronic and integrated optical CNN implementation for edge devices or cellphones, providing them with practical and powerful edge computing abilities and facilitating diverse applications, such as intelligent robotics, industrial automation, medical diagnosis, and remote sensing.

Optics-free imaging of complex, non-sparse and color QR-codes with deep neural networks

Optics-free imaging of complex, non-sparse QR-codes with Deep Neural Networks

Classification of optics-free images with deep neural networks

Transferable polychromatic optical encoder for neural networks

Spatially Varying Nanophotonic Neural Networks

Image sensing with multilayer nonlinear optical neural networks

Ciphertext only attack on QR code optical encryption system with spatially incoherent illumination using a neural network

Lensless Opto-Electronic Neural Network with Quantum Dot Nonlinear Activation

Ultrafast machine vision with 2D material neural network image sensors

Lensless opto-electronic neural network with quantum dots nonlinear activation

Compressed Meta-Optical Encoder for Image Classification

Free-space optical neural network based on thermal atomic nonlinearity

Spectral Convolutional Neural Network Chip for In-sensor Edge Computing of Incoherent Natural Light

Neuro-inspired optical sensor array for high-accuracy static image recognition and dynamic trace extraction

Neural nano-optics for high-quality thin lens imaging

Pre-sensor Computing with Compact Multilayer Optical Neural Network

Optical Neural Networks for Holographic Image Recognition (invited Paper)

An Optical Frontend for a Convolutional Neural Network

Single-shot optical neural network

Convolutional neural network advances in demosaicing for fluorescent cancer imaging with color-near-infrared sensors

EgoQR: Efficient QR Code Reading in Egocentric Settings