Memory-less scattering imaging with ultrafast convolutional optical neural networks

Yuchao Zhang,Qiming Zhang,Haoyi Yu,Yinan Zhang,Haitao Luan,Min Gu

DOI: https://doi.org/10.1126/sciadv.adn2205

IF: 13.6

2024-06-16

Science Advances

Abstract:The optical memory effect in complex scattering media including turbid tissue and speckle layers has been a critical foundation for macroscopic and microscopic imaging methods. However, image reconstruction from strong scattering media without the optical memory effect has not been achieved. Here, we demonstrate image reconstruction through scattering layers where no optical memory effect exists, by developing a multistage convolutional optical neural network (ONN) integrated with multiple parallel kernels operating at the speed of light. Training this Fourier optics-based, parallel, one-step convolutional ONN with the strong scattering process for direct feature extraction, we achieve memory-less image reconstruction with a field of view enlarged by a factor up to 271. This device is dynamically reconfigurable for ultrafast multitask image reconstruction with a computational power of 1.57 peta-operations per second (POPS). Our achievement establishes an ultrafast and high energy-efficient optical machine learning platform for graphic processing.

multidisciplinary sciences

What problem does this paper attempt to address?

This paper addresses the problem of image reconstruction in strongly scattering media without the optical memory effect. Traditional optical imaging techniques rely on the optical memory effect, which is the correlation of scattered photons under minute changes, but this effect becomes negligible in complex scattering layers. The research team developed a multi-stage convolutional optical neural network (ONN) that combines multiple parallel kernels, operates at the speed of light, and is capable of image reconstruction without the optical memory effect. This approach expands the field of view, enhances reconstruction speed and energy efficiency, and is suitable for graphic processing, as well as being able to dynamically reconstruct for ultrafast multitask image reconstruction. By training a parallel one-step convolutional ONN based on Fourier optics, they were able to directly extract features in scattering layers without the optical memory effect, achieving image reconstruction of hidden objects.

Memory-less scattering imaging with ultrafast convolutional optical neural networks

Electrically Programmable Phase-Change Photonic Memory for Optical Neural Networks with Nanoseconds in Situ Training Capability

Non-volatile and Ultra-Fast Photonic Vector Accelerator with Optical Phase Change Materials and Integrated Microcomb

OP-FCNN: an Optronic Fully Convolutional Neural Network for Imaging Through Scattering Media.

On-chip 4F-System Based on Concave Mirrors for Optical Neural Networks

11 TeraFLOPs per second photonic convolutional accelerator for deep learning optical neural networks

Optical neuromorphic processing at Tera-OP/s speeds based on Kerr soliton crystal microcombs

Sophisticated Deep Learning with On-Chip Optical Diffractive Tensor Processing

11 Tera-OPs/s Photonic Convolutional Accelerator and Deep Optical Neural Network Based on an Integrated Kerr Soliton Crystal Microcomb

Learning-based Method to Reconstruct Complex Targets Through Scattering Medium Beyond the Memory Effect.

Scalable optical neural networks based on temporal computing

A Self-Attention Armed Optronic Transformer in Imaging Through Scattering Media

11 TOPS photonic convolutional accelerator for optical neural networks

Non-invasive imaging through scattering medium and around corners beyond 3D memory effect.

Learning-based real-time method to looking through scattering medium oriented to practical application

New Residual Neural Network for Rapid Imaging of Scattering

Optical neuromorphic processing with soliton crystal Kerr microcombs: Scaling the network in size and speed

Integrated Photonic FFT for Optical Convolutions towards Efficient and High-Speed Neural Networks

Spatially Varying Nanophotonic Neural Networks

High‐Resolution Optical Convolutional Neural Networks Using Phase‐Change Material‐Based Microring Hybrid Waveguides

All-Optical Computing Based on Convolutional Neural Networks