An optical imager that can compute

Zheng Huang,Hongwei Chen

DOI: https://doi.org/10.1038/s41565-023-01590-1

IF: 38.3

2024-02-06

Nature Nanotechnology

Abstract:An intelligent meta-imager can perform multi-channel convolutions to accelerate machine vision for incoherent light.

materials science, multidisciplinary,nanoscience & nanotechnology

What problem does this paper attempt to address?

A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture

Ruibing Song,Kejie Huang,Zongsheng Wang,Haibin Shen

DOI: https://doi.org/10.1109/tcsvt.2022.3179370

IF: 5.859

2022-01-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:The separation of the data capture and analysis in modern vision systems has led to a massive amount of data transfer between the end devices and cloud computers, resulting in long latency, slow response, and high power consumption. Efficient hardware architectures are under focused development to enable Artificial Intelligence (AI) at the resource-limited sensing devices. One of the most promising solutions is to enable Processing-in-Pixel (PIP) scheme. However, the conventional schemes suffer from the low fill-factor issue. This paper proposes a PIP based Complementary Metal-Oxide-Semiconductor (CMOS) sensor architecture, which allows convolution operation before the column readout circuit to significantly reduce the overall power consumption while improving the resource utilization of the succeeding deep learning accelerator. The simulation results show that the proposed architecture could support the computing efficiency up to 3.37 TOPS/W at the 8-bit weight configuration, which is four times as high as the conventional schemes after normalization. The transistors required for each pixel are only 3.5T, significantly improving the fill-factor.
Intelligent Multi-channel Meta-imagers for Accelerating Machine Vision

Hanyu Zheng,Quan Liu,Ivan I. Kravchenko,Xiaomeng Zhang,Yuankai Huo,Jason G. Valentine

2023-06-13

Abstract:Rapid developments in machine vision have led to advances in a variety of industries, from medical image analysis to autonomous systems. These achievements, however, typically necessitate digital neural networks with heavy computational requirements, which are limited by high energy consumption and further hinder real-time decision-making when computation resources are not accessible. Here, we demonstrate an intelligent meta-imager that is designed to work in concert with a digital back-end to off-load computationally expensive convolution operations into high-speed and low-power optics. In this architecture, metasurfaces enable both angle and polarization multiplexing to create multiple information channels that perform positive and negatively valued convolution operations in a single shot. The meta-imager is employed for object classification, experimentally achieving 98.6% accurate classification of handwritten digits and 88.8% accuracy in classifying fashion images. With compactness, high speed, and low power consumption, this approach could find a wide range of applications in artificial intelligence and machine vision applications.

Computer Vision and Pattern Recognition,Optics
Multichannel meta-imagers for accelerating machine vision

Hanyu Zheng,Quan Liu,Ivan I. Kravchenko,Xiaomeng Zhang,Yuankai Huo,Jason G. Valentine

DOI: https://doi.org/10.1038/s41565-023-01557-2

IF: 38.3

2024-01-05

Nature Nanotechnology

Abstract:Rapid developments in machine vision technology have impacted a variety of applications, such as medical devices and autonomous driving systems. These achievements, however, typically necessitate digital neural networks with the downside of heavy computational requirements and consequent high energy consumption. As a result, real-time decision-making is hindered when computational resources are not readily accessible. Here we report a meta-imager designed to work together with a digital back end to offload computationally expensive convolution operations into high-speed, low-power optics. In this architecture, metasurfaces enable both angle and polarization multiplexing to create multiple information channels that perform positively and negatively valued convolution operations in a single shot. We use our meta-imager for object classification, achieving 98.6% accuracy in handwritten digits and 88.8% accuracy in fashion images. Owing to its compactness, high speed and low power consumption, our approach could find a wide range of applications in artificial intelligence and machine vision applications.

materials science, multidisciplinary,nanoscience & nanotechnology
Optical Convolution Based Computational Method for Low-Power Image Processing

Yi Huang,Yuan Chen,Zhentao Qin,Jinlei Zhang,Zhenrong Zheng

DOI: https://doi.org/10.1117/12.2527733

2019-01-01

Abstract:Convolutional neural networks (CNNs) is becoming a critical role for deep learning-based computer vision applications. Through CNN we can extract meaningful information out of massive sensor data. Vision applications use this information to analyze the mainstream trends of the data and take immediate action based on these trends. However, CNN's energy consumption and bandwidth limitations make it difficult for CNN network systems to deploy in mobile systems with stringent energy limits. In this paper, we explore the simulation and the hardware method of optical convolution for low power image processing, which is inspired by the bio-image sensor Angle sensitive pixel (ASP). This optical computation method may be used to substitute the first convolution layer of the CNN network due to its energy-saving features and speed of light processing time. We adopt two-layer of customized transmission grating to perform this optical convolution computing. By generating the Talbot effect, the two-layer grating structure can perform optical convolution computation using Gabor wavelet filters, which will cause zero electrical power. We demonstrate both simulation and experiment results for optical convolution through our algorithm and prototype system, the convolution results can extract different meaningful information about the original image, which is very similar to edge filtering. This optical operation will hopefully be used to replace the first convolution layer of CNN since it can effectively reduce both the consumption of the energy power and the performing time.
A Multimodal Sensing CMOS Imager Based on Dual‐Focus Imaging(Adv. Sci. 14/2023)

Hao Dong,Xubin Zheng,Chen Cheng,Libin Qian,Yaoxuan Cui,Weiwei Wu,Qingjun Liu,Xing Chen,Yanli Lu,Qing Yang,Fenni Zhang,Di Wang

DOI: https://doi.org/10.1002/advs.202206699

IF: 15.1

2023-01-01

Advanced Science

Abstract:Advanced machine intelligence is empowered not only by the ever-increasing computational capability for information processing but also by sensors for collecting multimodal information from complex environments. However, simply assembling different sensors can result in bulky systems and complex data processing. Herein, it is shown that a complementary metal-oxide-semiconductor (CMOS) imager can be transformed into a compact multimodal sensing platform through dual-focus imaging. By combining lens-based and lensless imaging, visual information, chemicals, temperature, and humidity can be detected with the same chip and output as a single image. As a proof of concept, the sensor is equipped on a micro-vehicle, and multimodal environmental sensing and mapping is demonstrated. A multimodal endoscope is also developed, and simultaneous imaging and chemical profiling along a porcine digestive tract is achieved. The multimodal CMOS imager is compact, versatile, and extensible and can be widely applied in microrobots, in vivo medical apparatuses, and other microdevices.
A multichannel optical computing architecture for advanced machine vision

Zhihao Xu,Xiaoyun Yuan,Tiankuang Zhou,Lu Fang

DOI: https://doi.org/10.1038/s41377-022-00945-y

2022-08-19

Guangxue Jishu/Optical Technique

Abstract:Abstract Endowed with the superior computing speed and energy efficiency, optical neural networks (ONNs) have attracted ever-growing attention in recent years. Existing optical computing architectures are mainly single-channel due to the lack of advanced optical connection and interaction operators, solving simple tasks such as hand-written digit classification, saliency detection, etc. The limited computing capacity and scalability of single-channel ONNs restrict the optical implementation of advanced machine vision. Herein, we develop Monet: a multichannel optical neural network architecture for a universal multiple-input multiple-channel optical computing based on a novel projection-interference-prediction framework where the inter- and intra- channel connections are mapped to optical interference and diffraction. In our Monet, optical interference patterns are generated by projecting and interfering the multichannel inputs in a shared domain. These patterns encoding the correspondences together with feature embeddings are iteratively produced through the projection-interference process to predict the final output optically. For the first time, Monet validates that multichannel processing properties can be optically implemented with high-efficiency, enabling real-world intelligent multichannel-processing tasks solved via optical computing, including 3D/motion detections. Extensive experiments on different scenarios demonstrate the effectiveness of Monet in handling advanced machine vision tasks with comparative accuracy as the electronic counterparts yet achieving a ten-fold improvement in computing efficiency. For intelligent computing, the trends of dealing with real-world advanced tasks are irreversible. Breaking the capacity and scalability limitations of single-channel ONN and further exploring the multichannel processing potential of wave optics, we anticipate that the proposed technique will accelerate the development of more powerful optical AI as critical support for modern advanced machine vision.
Integration of Communication and Computational Imaging

Zhenming Yu,Liming Cheng,Hongyu Huang,Wei Zhang,Liang Lin,Kun Xu

2024-10-29

Abstract:Communication enables the expansion of human visual perception beyond the limitations of time and distance, while computational imaging overcomes the constraints of depth and breadth. Although impressive achievements have been witnessed with the two types of technologies, the occlusive information flow between the two domains is a bottleneck hindering their ulterior progression. Herein, we propose a novel framework that integrates communication and computational imaging (ICCI) to break through the inherent isolation between communication and computational imaging for remote perception. By jointly considering the sensing and transmitting of remote visual information, the ICCI framework performs a full-link information transfer optimization, aiming to minimize information loss from the generation of the information source to the execution of the final vision tasks. We conduct numerical analysis and experiments to demonstrate the ICCI framework by integrating communication systems and snapshot compressive imaging systems. Compared with straightforward combination schemes, which sequentially execute sensing and transmitting, the ICCI scheme shows greater robustness against channel noise and impairments while achieving higher data compression. Moreover, an 80 km 27-band hyperspectral video perception with a rate of 30 fps is experimentally achieved. This new ICCI remote perception paradigm offers a highefficiency solution for various real-time computer vision tasks.

Image and Video Processing,Computer Vision and Pattern Recognition,Signal Processing
Optical Multi-Imaging-Casting Accelerator for Fully Parallel Universal Convolution Computing

Changhe Zhou,Guoqing Ma,Rongwei Zhu,Junjie Yu

DOI: https://doi.org/10.21203/rs.3.rs-870558/v1

2021-10-11

Abstract:Abstract Over the last few years, optical computing has become a potential solution to computationally heavy convolution, aimed at accelerating various artificial intelligence applications. However, past schemes have never efficiently realized fully parallel optical convolution. Here, we propose a new paradigm for a universal convolution accelerator with truly massive parallelism and high precision based on optical multi-imaging-casting architecture. Specifically, a two-dimensional Dammann grating is adopted for the generation of multiple displaced images of the kernel, which is the core process for kernel sliding on the convolved matrix. Our experimental results indicate that the computing accuracy is typically close to 8-bit, and this accuracy can be improved further by using hybrid analog–digital coding method. In addition, a convolutional neural network for the standard MNIST dataset is demonstrated, and the recognition accuracy for inference is up to 97.3%. The paradigm reported here will open new opportunities for high-throughput universal convolution accelerators for real-time or quasi-real-time AI applications.
A multichannel optical neural network for advanced machine vision

Lu Fang,Zhihao Xu,Xiaoyun Yuan,Tiankuang Zhou

DOI: https://doi.org/10.21203/rs.3.rs-1283910/v1

2022-01-01

Abstract:Abstract Endowed with the superior computing speed and energy efficiency, all-optical and optoelectronics neural networks (ONNs) have attracted ever-growing attention in recent years. Previous studies lacking viable way for multichannel optical processing, mainly implement single-channel optical computing to solve simple tasks including hand-written digit classification, saliency detection, etc. Aiming for more powerful ONNs to solve complex tasks, we innovatively develop Monet: a multichannel optical neural network for advanced machine vision. The inter- and intra- channel connections are mapped to optical interference and diffraction. By further designing a novel projection-interference-prediction framework on the basis of these connections, optical interference patterns are generated by projecting and interfering the multichannel inputs in a shared domain. These patterns encoding the correspondences together with feature embeddings are iteratively produced through the projection-interference process to predict the final output. Unlike existing ONNs trying to inherit ANN architectures, the framework of Monet is initially designed following the innate characters of light propagation and interaction, to fully explore the potential of wave optics for optical computing. For the first time, we validate that Monet is capable for advanced machine vision tasks such as 3D depth estimation and moving objection detection. Moreover, a prototype system of Monet is developed to accomplish 3D perception for real-world scenarios. We anticipate that the proposed technique will accelerate the development of more powerful optical AI as critical support for modern advanced machine vision.
Resource-Saving and High-Robustness Image Sensing Based on Binary Optical Computing

Zhanhong Zhou,Ziwei Li,Wei Zhou,Nan Chi,Junwen Zhang,Qionghai Dai

DOI: https://doi.org/10.1002/lpor.202400936

2024-01-01

LASER & PHOTONICS REVIEWS

Abstract:Computational imaging, as a novel technology utilizing encoded image acquisition, relies on intelligent decoding methods for effective image restoration and sensing. Optical computing-based decoders can efficiently process and extract features from pre-sensor information, reducing the computational burden on digital computers. However, mainstream parallel optical neural network (ONN) architectures based on wavefront propagation typically possess complex network structures and high-precision parameters, which pose challenges in terms of precise fabrication and system calibration, as well as sensitivity to signal-to-noise ratios. In this work, a binary-weighted optical computing engine is proposed with spatial multiplexing and aggregation (B-OSMA), a large-scale passive ONN implementation that achieves high-efficiency image sensing. Employing B-OSMA as an optical decoder, demonstrated image categorizing from 2% compressive is experimented sampling with 92.0% and 83.8% accuracy on MNIST and fashion-MNIST datasets, respectively, approaching the performance of full-precision electronic computing while reducing storage requirements by 97%. Compared to conventional ONNs with analog weights, the B-OSMA exhibits enhanced resilience against systematic errors and ambient noise. This work represents a significant advancement towards practical applications of optical computing in image sensing. A novel binary-weighted optical computing implementation with spatial multiplexing and aggregation (B-OSMA) scheme achieves high-efficiency image sensing. Coupling with a light-weighted digital discriminator, B-OSMA demonstrates good performances that are competitive to full-precision electric computing on MNIST and Fashion-MNIST datasets, with greatly reduced storage consumption and promoted resilience against systematic errors and ambient noise. image
CoCoCs: Co-Optimized Compressive Imaging Driven by High-Level Vision

Honghao Huang,Chengyang Hu,Jingwei Li,Xiaowen Dong,Hongwei Chen

DOI: https://doi.org/10.1364/oe.468733

IF: 3.8

2022-01-01

Optics Express

Abstract:Compressive imaging senses optically encoded high-dimensional scene data with far fewer measurements and then performs reconstruction via appropriate algorithms. In this paper, we present a novel noniterative end-to-end deep learning-based framework for compressive imaging, dubbed CoCoCs. In comparison to existing approaches, we extend the pipeline by co-optimizing the recovery algorithm with optical coding as well as cascaded high-level computer vision tasks to boost the quality of the reconstruction. We demonstrate the proposed framework on two typical compressive imaging systems, i.e., single pixel imaging and snapshot video compressive imaging. Extensive results, including conventional image quality criteria, mean opinion scores, and accuracy in image classification and motion recognition, confirm that CoCoCs can yield realistic images and videos, which are friendly to both human viewing and computer vision. We hope CoCoCs will give impetus to bridge the gap between compressive imagers and computer vision and the perception of human.
11 TeraFLOPs per second photonic convolutional accelerator for deep learning optical neural networks

Xingyuan Xu,Mengxi Tan,Bill Corcoran,Jiayang Wu,Andreas Boes,Thach G. Nguyen,Sai T. Chu,Brent E. Little,Damien G. Hicks,Roberto Morandotti,Arnan Mitchell,David J. Moss

DOI: https://doi.org/10.1038/s41586-020-03063-0

2020-11-15

Abstract:Convolutional neural networks (CNNs), inspired by biological visual cortex systems, are a powerful category of artificial neural networks that can extract the hierarchical features of raw data to greatly reduce the network parametric complexity and enhance the predicting accuracy. They are of significant interest for machine learning tasks such as computer vision, speech recognition, playing board games and medical diagnosis. Optical neural networks offer the promise of dramatically accelerating computing speed to overcome the inherent bandwidth bottleneck of electronics. Here, we demonstrate a universal optical vector convolutional accelerator operating beyond 10 TeraFLOPS (floating point operations per second), generating convolutions of images of 250,000 pixels with 8 bit resolution for 10 kernels simultaneously, enough for facial image recognition. We then use the same hardware to sequentially form a deep optical CNN with ten output neurons, achieving successful recognition of full 10 digits with 900 pixel handwritten digit images with 88% accuracy. Our results are based on simultaneously interleaving temporal, wavelength and spatial dimensions enabled by an integrated microcomb source. This approach is scalable and trainable to much more complex networks for demanding applications such as unmanned vehicle and real-time video recognition.

Neural and Evolutionary Computing,Applied Physics,Optics
An optical biomimetic eyes with interested object imaging

Jun Li,Shimei Chen,Shangyuan Wang,Miao Lei,Xiaofang Dai,Chuangxue Liang,Kunyuan Xu,Shuxin Lin,Yuhui Li,Yuer Fan,Ting Zhong

DOI: https://doi.org/10.48550/arXiv.2108.04236

2021-08-08

Abstract:We presented an optical system to perform imaging interested objects in complex scenes, like the creature easy see the interested prey in the hunt for complex environments. It utilized Deep-learning network to learn the interested objects's vision features and designed the corresponding "imaging matrices", furthermore the learned matrixes act as the measurement matrix to complete compressive imaging with a single-pixel camera, finally we can using the compressed image data to only image the interested objects without the rest objects and backgrounds of the scenes with the previous Deep-learning network. Our results demonstrate that no matter interested object is single feature or rich details, the interference can be successfully filtered out and this idea can be applied in some common applications that effectively improve the performance. This bio-inspired optical system can act as the creature eye to achieve success on interested-based object imaging, object detection, object recognition and object tracking, etc.

Computer Vision and Pattern Recognition,Optics
Intelligent Meta-Imagers: From Compressed to Learned Sensing

Chloé Saigre-Tardif,Rashid Faqiri,Hanting Zhao,Lianlin Li,Philipp del Hougne

DOI: https://doi.org/10.1063/5.0076022

2022-01-29

Abstract:Computational meta-imagers synergize metamaterial hardware with advanced signal processing approaches such as compressed sensing. Recent advances in artificial intelligence (AI) are gradually reshaping the landscape of meta-imaging. Most recent works use AI for data analysis, but some also use it to program the physical meta-hardware. The role of "intelligence" in the measurement process and its implications for critical metrics like latency are often not immediately clear. Here, we comprehensively review the evolution of computational meta-imaging from the earliest frequency-diverse compressive systems to modern programmable intelligent meta-imagers. We introduce a clear taxonomy in terms of the flow of task-relevant information that has direct links to information theory: compressive meta-imagers indiscriminately acquire all scene information in a task-agnostic measurement process that aims at a near-isometric embedding; intelligent meta-imagers highlight task-relevant information in a task-aware measurement process that is purposefully non-isometric. The measurement process of intelligent meta-imagers is thus simultaneously an analog wave processor that implements a first task-specific inference step "over-the-air". We provide explicit design tutorials for the integration of programmable meta-atoms as trainable physical weights into an intelligent end-to-end sensing pipeline. This merging of the physical world of metamaterial engineering and the digital world of AI enables the remarkable latency gains of intelligent meta-imagers. We further outline emerging opportunities for cognitive meta-imagers with reverberation-enhanced resolution and we point out how the meta-imaging community can reap recent advances in the vibrant field of metamaterial wave processors to reach the holy grail of low-energy ultra-fast all-analog intelligent meta-sensors.

Applied Physics,Information Theory,Signal Processing,Systems and Control,Optics
11 Tera-OPs/s Photonic Convolutional Accelerator and Deep Optical Neural Network Based on an Integrated Kerr Soliton Crystal Microcomb

Mengxi Tan,Xingyuan Xu,Yang Li,Yang Sun,Damien Hicks,Roberto Morandotti,Jiayang Wu,Arnan Mitchell,David J. Moss

DOI: https://doi.org/10.1117/12.2607906

2022-01-01

Abstract:Convolutional neural networks (CNNs), inspired by biological visual cortex systems, are a powerful category of artificial neural networks that can extract the hierarchical features of raw data to greatly reduce the network parametric complexity and enhance the predicting accuracy. They are of significant interest for machine learning tasks such as computer vision, speech recognition, playing board games and medical diagnosis [1-7]. Optical neural networks offer the promise of dramatically accelerating computing speed to overcome the inherent bandwidth bottleneck of electronics. Here, we demonstrate a universal optical vector convolutional accelerator operating beyond 10 Tera-OPS (TOPS operations per second), generating convolutions of images of 250,000 pixels with 8-bit resolution for 10 kernels simultaneously - enough for facial image recognition. We then use the same hardware to sequentially form a deep optical CNN with ten output neurons, achieving successful recognition of full 10 digits with 900 pixel handwritten digit images with 88% accuracy. Our results are based on simultaneously interleaving temporal, wavelength and spatial dimensions enabled by an integrated microcomb source. We show that this approach is scalable and trainable to much more complex networks for demanding applications such as unmanned vehicle and real-time video recognition.
Slimmable Multi-Task Image Compression for Human and Machine Vision

Jiangzhong Cao,Ximei Yao,Huan Zhang,Jian Jin,Yun Zhang,Bingo Wing-Kuen Ling

DOI: https://doi.org/10.1109/access.2023.3261668

IF: 3.9

2023-01-01

IEEE Access

Abstract:In the Internet of Things (IoT) communications, visual data are frequently processed among intelligent devices using artificial intelligence algorithms, replacing humans for analysis and decision-making while only occasionally requiring human scrutiny. However, due to high redundancy of compressive encoders, existing image coding solutions for machine vision are inefficient at runtime. To balance the rate-accuracy performance and efficiency of image compression for machine vision while attaining high-quality reconstructed images for human vision, this paper introduces a novel slimmable multi-task compression framework for human and machine vision in visual IoT applications. Firstly, image compression for human and machine vision under the constraint of bandwidth, latency, and computational resources is modeled as a multi-task optimization problem. Secondly, slimmable encoders are employed for multiple human and machine vision tasks in which the parameters of the sub-encoder for machine vision tasks are shared among all tasks and jointly learned. Thirdly, to solve the feature match between latent representation and intermediate features of deep vision networks, feature transformation networks are introduced as decoders of machine vision feature compression. Finally, the proposed framework is successfully applied to human and machine vision tasks’ scenarios, e.g., object detection and image reconstruction. Experimental results show that the proposed method outperforms baselines and other image compression approaches on machine vision tasks with higher efficiency (shorter latency) in two vision tasks’ scenarios while retaining comparable quality on image reconstruction.

computer science, information systems,telecommunications,engineering, electrical & electronic
Differentiable Imaging: A New Tool for Computational Optical Imaging

Ni Chen,Liangcai Cao,Ting-Chung Poon,Byoungho Lee,Edmund Y. Lam

DOI: https://doi.org/10.1002/apxr.202200118

2023-01-01

Abstract:The field of computational imaging has made significant advancements in recent years, yet it still faces limitations due to the restrictions imposed by traditional computational techniques. Differentiable programming offers a solution by combining the strengths of classical optimization and deep learning, enabling the creation of interpretable model‐based neural networks. Through the integration of physics into the modeling process, differentiable imaging, which employs differentiable programming in computational imaging, has the potential to overcome challenges posed by sparse, incomplete, and noisy data. As a result, it has the potential to play a key role in advancing the field of computational imaging and its various applications.
Optical Convolutional Neural Networks -- Combining Silicon Photonics and Fourier Optics for Computer Vision

Edward Cottle,Florent Michel,Joseph Wilson,Nick New,Iman Kundu

DOI: https://doi.org/10.48550/arXiv.2103.09044

2020-12-19

Image and Video Processing

Abstract:The Convolutional Neural Network (CNN) is a state-of-the-art architecture for a wide range of deep learning problems, the quintessential example of which is computer vision. CNNs principally employ the convolution operation, which can be accelerated using the Fourier transform. In this paper, we present an optical hardware accelerator that combines silicon photonics and free-space optics, leveraging the use of the optical Fourier transform within several CNN architectures. The hardware presented is a proof of concept, demonstrating that this technology can be applied to artificial intelligence problems with a large efficiency boost with respect to canonical methods.
Split-Aperture 2-in-1 Computational Cameras

Zheng Shi,Ilya Chugunov,Mario Bijelic,Geoffroi Côté,Jiwoon Yeom,Qiang Fu,Hadi Amata,Wolfgang Heidrich,Felix Heide

DOI: https://doi.org/10.1145/3658225

IF: 6.2

2024-07-19

ACM Transactions on Graphics

Abstract:While conventional cameras offer versatility for applications ranging from amateur photography to autonomous driving, computational cameras allow for domain-specific adaption. Cameras with co-designed optics and image processing algorithms enable high-dynamic-range image recovery, depth estimation, and hyperspectral imaging through optically encoding scene information that is otherwise undetected by conventional cameras. However, this optical encoding creates a challenging inverse reconstruction problem for conventional image recovery, and often lowers the overall photographic quality. Thus computational cameras with domain-specific optics have only been adopted in a few specialized applications where the captured information cannot be acquired in other ways. In this work, we investigate a method that combines two optical systems into one to tackle this challenge. We split the aperture of a conventional camera into two halves: one which applies an application-specific modulation to the incident light via a diffractive optical element to produce a coded image capture, and one which applies no modulation to produce a conventional image capture. Co-designing the phase modulation of the split aperture with a dual-pixel sensor allows us to simultaneously capture these coded and uncoded images without increasing physical or computational footprint. With an uncoded conventional image alongside the optically coded image in hand, we investigate image reconstruction methods that are conditioned on the conventional image, making it possible to eliminate artifacts and compute costs that existing methods struggle with. We assess the proposed method with 2-in-1 cameras for optical high-dynamic-range reconstruction, monocular depth estimation, and hyperspectral imaging, comparing favorably to all tested methods in all applications.

computer science, software engineering
Multichannel meta-imagers based on electrically tunable metasurfaces for accelerating matrix operations

Mengguang Wang,Qiangbo Zhang,Xinyu Liu,Chang Wang,Zeqing Yu,Bingliang Chen,Zhenrong Zheng

DOI: https://doi.org/10.1364/oe.537622

IF: 3.8

2024-10-19

Optics Express

Abstract:Mengguang Wang, Qiangbo Zhang, Xinyu Liu, Chang Wang, Zeqing Yu, Bingliang Chen, Zhenrong Zheng Optical neural networks have facilitated groundbreaking advancements in machine vision, intelligent healthcare, and autonomous driving ... [Opt. Express 32, 39915-39923 (2024)]

optics

An optical imager that can compute

A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture

Intelligent Multi-channel Meta-imagers for Accelerating Machine Vision

Multichannel meta-imagers for accelerating machine vision

Optical Convolution Based Computational Method for Low-Power Image Processing

A Multimodal Sensing CMOS Imager Based on Dual‐Focus Imaging(Adv. Sci. 14/2023)

A multichannel optical computing architecture for advanced machine vision

Integration of Communication and Computational Imaging

Optical Multi-Imaging-Casting Accelerator for Fully Parallel Universal Convolution Computing

A multichannel optical neural network for advanced machine vision

Resource-Saving and High-Robustness Image Sensing Based on Binary Optical Computing

CoCoCs: Co-Optimized Compressive Imaging Driven by High-Level Vision

11 TeraFLOPs per second photonic convolutional accelerator for deep learning optical neural networks

An optical biomimetic eyes with interested object imaging

Intelligent Meta-Imagers: From Compressed to Learned Sensing

11 Tera-OPs/s Photonic Convolutional Accelerator and Deep Optical Neural Network Based on an Integrated Kerr Soliton Crystal Microcomb

Slimmable Multi-Task Image Compression for Human and Machine Vision

Differentiable Imaging: A New Tool for Computational Optical Imaging

Optical Convolutional Neural Networks -- Combining Silicon Photonics and Fourier Optics for Computer Vision

Split-Aperture 2-in-1 Computational Cameras

Multichannel meta-imagers based on electrically tunable metasurfaces for accelerating matrix operations