Abstract:The advent of big data and the Internet of Things has created urgent demands for in-sensor computing hardware with multimodal perception that can effectively resolve the inefficiency, high latency, and excessive energy consumption challenges faced by conventional sensory systems. Here, a simple-structured optoelectronic synaptic device of In2O3SnO2/Nb:SrTiO3 (ITO/NSTO) heterostructure is proposed, which vividly demonstrates in-sensor computing and multimodal perception capabilities. First, with the ingenious synaptic responses of the device under both optical and electrical stimuli, a multimodal in-sensor neuromorphic computing system capable of concurrently perceiving and processing visual and auditory information is constructed. Using this multimodal system to perform a human emotion recognition task, the misjudgment arising from single-modal cognition can thereby be effectively avoided. Second, utilizing the integrated sensing and processing functions, along with the dynamic memories of the device array, a neuromorphic vision system is implemented for real-time monitoring of moving vehicles, which displays high recognition efficiency and accuracy. This research not only provides an optoelectronic synaptic device that is low-cost and easy to mass-produce, but also paves the way for next-generation in-sensor computing that can efficiently process multimodal signals. The separated sensing, storing, and computing modules along with single-mode perception in the conventional sensory system cause time-delay, massive energy consumption, as well as incomprehensive cognition issues. Here a neuromorphic device capable of performing multimodal perception and in-sensor computation within a single device is proposed, enabling a novel multimodal sensory system that is efficient, low-energy, and has minimal time latency. image

A Multimodal Sensing CMOS Imager Based on Dual‐Focus Imaging

A Multimodal Sensing CMOS Imager Based on Dual‐Focus Imaging(Adv. Sci. 14/2023)

Multiplexed Chemical Sensing CMOS Imager.

Optical Micro/nanofibre Embedded Soft Film Enables Multifunctional Flow Sensing in Microfluidic Chips

Metalens-integrated Compact Imaging Devices for Wide-Field Microscopy

Multi-Channel Light Addressable Potentiometric Sensor Based on Microlens Array

Multimodal In‐Sensor Computing System Using Integrated Silicon Photonic Convolutional Processor

Multimodal In-Sensor Computing Implemented by Easily-Fabricated Oxide-Heterojunction Optoelectronic Synapses

Microimage Processing System Based On Cmos Sensor

Portable Multi-Spectral Lens-Less Microscope with Wavelength-Self-calibrating Imaging Sensor

Design of Optoelectronic Imaging System with High Resolution and Large Field of View Based on Dual Cmos

A Multimodal and Multifunctional CMOS Cellular Interfacing Array for Digital Physiology and Pathology Featuring an Ultra Dense Pixel Array and Reconfigurable Sampling Rate

(Invited) CMOS-Based Multimodal Sensing

Ultracompact Multimode Meta-Microscope Based on Both Spatial and Guided-Wave Illumination

Multi-Mode Microscopic Hyperspectral Imager For The Sensing Of Biological Samples

Silicon Nanomembrane Phototransistor Flipped with Multifunctional Sensors Toward Smart Digital Dust

DUAL-MODE HYPERSPECTRAL BIO-IMAGER WITH A CONJUGATED CAMERA FOR QUICK OBJECT-SELECTION AND FOCUSING

Metaoptronic Multiplexed Interface for Probing Bioentity Behaviors

High Numerical Aperture Imaging Systems Formed by Integrating Bionic Artificial Compound Eyes on a CMOS Sensor

Miniature Endoscope for Multimodal Imaging

CompdVision: Combining Near-Field 3D Visual and Tactile Sensing Using a Compact Compound-Eye Imaging System