Abstract:Convolutional neural networks (CNNs) are representative models of artificial neural networks (ANNs). However, the considerable power consumption and limited computing speed of electrical computing platforms restrict further CNN development on edge devices. Optical neural networks are considered next-generation physical implementations of ANNs, but their capabilities are limited by on-chip integration scale and requirement for coherent light sources. This study proposes a spectral convolutional neural network (SCNN) of incoherent natural light by an optical convolutional layer (OCL) and a reconfigurable electrical backend. The OCL is implemented by integrating very large-scale, pixel-aligned spectral filters on a CMOS image sensor on a 12-inch wafer, facilitating highly parallel spectral vector-inner products of incident light. It accepts broadband incoherent natural light containing two spatial and one spectral dimension directly as input with the function of matter meta-imaging. This unique optoelectronic framework empowers in-sensor optical analog computing at extremely high energy efficiency because the OCL is driven by the energy of the information carrier, i. e. natural light. To the best of our knowledge, this is the first integrated optical computing utilizing natural light. We employ the same SCNN chip for completely different real-world complex tasks,and achieve accuracies of over 96% for pathological diagnosis and almost 100% for face anti-spoofing at video rates. The SCNN framework has an unprecedented new function of substance identification, provides a feasible optoelectronic and integrated optical CNN implementation for edge devices or cellphones, providing them with practical and powerful edge computing abilities and facilitating diverse applications, such as intelligent robotics, industrial automation, medical diagnosis, and remote sensing.

Convolutional Neural Networks with Coherent Nanophotonic Circuits

Optical Convolution Based Computational Method for Low-Power Image Processing

Optical-electronic Hybrid Fourier Convolutional Neural Network Based on Super-Pixel Complex-Valued Modulation

A Convolution Neural Network Implemented by Three 3 × 3 Photonic Integrated Reconfigurable Linear Processors

Clements-Enhanced Complex-Valued Coherent Mesh with Balanced Detection Units for Photonic Neural Networks

Photonic Convolution Neural Network Based on Interleaved Time-Wavelength Modulation

On-chip 4F-System Based on Concave Mirrors for Optical Neural Networks

Spectral Convolutional Neural Network Chip for In-sensor Edge Computing of Incoherent Natural Light

Physical Layer-Aware Digital-Analog Co-Design for Photonic Convolution Neural Network

Integrated Photonic Convolutional Neural Network Based on Silicon Metalines

Optronic Convolutional Neural Networks of Multi-Layers with Different Functions Executed in Optics for Image Classification.

Optical Convolutional Neural Networks -- Combining Silicon Photonics and Fourier Optics for Computer Vision

11 Tera-OPs/s Photonic Convolutional Accelerator and Deep Optical Neural Network Based on an Integrated Kerr Soliton Crystal Microcomb

Highly efficient convolution computing architecture based on silicon photonic Fano resonance devices

Single chip photonic deep neural network with accelerated training

Zero-power Optical Convolutional Neural Network Using Incoherent Light

Diffraction‐Driven Parallel Convolution Processing with Integrated Photonics

High-energy-efficiency integrated photonic convolutional neural networks

Photonic Neural Networks and Its Applications

Electrically Programmable Phase-Change Photonic Memory for Optical Neural Networks with Nanoseconds in Situ Training Capability

Parallel Photonic Convolutional Processing On-Chip with Cross-Connect Architecture and Cyclic AWGs