Abstract:Optical imaging and sensing systems based on diffractive elements have seen massive advances over the last several decades. Earlier generations of diffractive optical processors were, in general, designed to deliver information to an independent system that was separately optimized, primarily driven by human vision or perception. With the recent advances in deep learning and digital neural networks, there have been efforts to establish diffractive processors that are jointly optimized with digital neural networks serving as their back-end. These jointly optimized hybrid (optical+digital) processors establish a new "diffractive language" between input electromagnetic waves that carry analog information and neural networks that process the digitized information at the back-end, providing the best of both worlds. Such hybrid designs can process spatially and temporally coherent, partially coherent, or incoherent input waves, providing universal coverage for any spatially varying set of point spread functions that can be optimized for a given task, executed in collaboration with digital neural networks. In this article, we highlight the utility of this exciting collaboration between engineered and programmed diffraction and digital neural networks for a diverse range of applications. We survey some of the major innovations enabled by the push-pull relationship between analog wave processing and digital neural networks, also covering the significant benefits that could be reaped through the synergy between these two complementary paradigms.

What problem does this paper attempt to address?

The paper primarily aims to address the following issues: 1. **Optimization of Optical Imaging and Sensing Systems**: The paper explores how to optimize optical imaging and sensing systems by combining programmable diffractive elements with digital neural networks (DNN). Traditional diffractive optical processors are usually designed to transmit information to an independent system, which is often optimized based on human vision or perception. With the development of deep learning and digital neural networks, researchers are attempting to establish a new "diffractive language" that allows the analog information carried by input electromagnetic waves to work in synergy with the backend digital neural network, thereby achieving optimal information processing results. 2. **Application of Hybrid Information Processing Technology**: By combining engineered diffraction with digital neural networks, it is possible to process spatially and temporally coherent, partially coherent, or incoherent input waves, providing a universal solution for various imaging and sensing tasks. This hybrid design not only leverages the advantages of photonics but also enhances overall performance with deep learning algorithms. 3. **Reconfigurable Applications**: The paper also discusses the collaboration between programmable diffraction and digital neural networks in the field of meta-optics. Specifically, researchers use deep learning techniques to solve the inverse problem of the codes required to control reconfigurable metasurfaces in real-time, enabling reconfigurable metasurfaces to play a greater role in practical applications. 4. **Computational Imaging**: By combining coded apertures with deep learning algorithms, various computational imaging tasks can be achieved, such as super-resolution imaging and extended depth-of-field imaging. This joint optimization approach often results in better performance than traditional methods. 5. **Acceleration of Information Encoding and Decoding**: Utilizing diffractive processing can accelerate the computation process of neural networks, especially when training and executing large deep neural networks. This method not only improves computation speed but also reduces power consumption requirements. In summary, this paper aims to demonstrate how combining advanced optical technologies with deep learning methods can address the limitations of traditional optical imaging and sensing systems and explore their potential in various practical application scenarios.

Integration of Programmable Diffraction with Digital Neural Networks

Design of task-specific optical systems using broadband diffractive neural networks

Scale-, shift- and rotation-invariant diffractive optical networks

Universal Linear Intensity Transformations Using Spatially-Incoherent Diffractive Processors

Time-lapse image classification using a diffractive neural network

In Situ Optical Backpropagation Training of Diffractive Optical Neural Networks

Deep Learning-designed Diffractive Neural Networks

<i>In situ</i> optical backpropagation training of diffractive optical neural networks

Simulating an Integrated Photonic Image Classifier for Diffractive Neural Networks

1-bit Quantized On-chip Hybrid Diffraction Neural Network Enabled by Authentic All-optical Fully-connected Architecture

A programmable diffractive deep neural network based on a digital-coding metasurface array

Solving Computer Vision Tasks with Diffractive Neural Networks

Advances and Progress of Diffractive Deep Neural Networks

Massively Parallel Universal Linear Transformations using a Wavelength-Multiplexed Diffractive Optical Network

Direct electromagnetic information processing with planar diffractive neural network

On-chip 4F-System Based on Concave Mirrors for Optical Neural Networks

Classification and reconstruction of spatially overlapping phase images using diffractive optical networks

Space-efficient optical computing with an integrated chip diffractive neural network

Nonlinear encoding in diffractive information processing using linear optical materials

Coherence Awareness in Diffractive Neural Networks

Scientific Computing with Diffractive Optical Neural Networks