Abstract:The advent of big data and the Internet of Things has created urgent demands for in-sensor computing hardware with multimodal perception that can effectively resolve the inefficiency, high latency, and excessive energy consumption challenges faced by conventional sensory systems. Here, a simple-structured optoelectronic synaptic device of In2O3SnO2/Nb:SrTiO3 (ITO/NSTO) heterostructure is proposed, which vividly demonstrates in-sensor computing and multimodal perception capabilities. First, with the ingenious synaptic responses of the device under both optical and electrical stimuli, a multimodal in-sensor neuromorphic computing system capable of concurrently perceiving and processing visual and auditory information is constructed. Using this multimodal system to perform a human emotion recognition task, the misjudgment arising from single-modal cognition can thereby be effectively avoided. Second, utilizing the integrated sensing and processing functions, along with the dynamic memories of the device array, a neuromorphic vision system is implemented for real-time monitoring of moving vehicles, which displays high recognition efficiency and accuracy. This research not only provides an optoelectronic synaptic device that is low-cost and easy to mass-produce, but also paves the way for next-generation in-sensor computing that can efficiently process multimodal signals. The separated sensing, storing, and computing modules along with single-mode perception in the conventional sensory system cause time-delay, massive energy consumption, as well as incomprehensive cognition issues. Here a neuromorphic device capable of performing multimodal perception and in-sensor computation within a single device is proposed, enabling a novel multimodal sensory system that is efficient, low-energy, and has minimal time latency. image

In-sensor computing for machine vision

Recent advances in in-sensor computational vision sensors: from mechanisms to applications

A Multimodal Sensing CMOS Imager Based on Dual‐Focus Imaging(Adv. Sci. 14/2023)

Review of bio-inspired image sensors for efficient machine vision

Bioinspired sensing-memory-computing integrated vision systems: biomimetic mechanisms, design principles, and applications

Artificial intelligence-based spatio-temporal vision sensors: applications and prospects

A Future Perspective on In-Sensor Computing

In-sensor Dynamic Computing for Intelligent Machine Vision

Ultrafast machine vision with 2D material neural network image sensors

Optoelectronic Devices for In-Sensor Computing

Near-sensor and in-sensor computing

Ultralow Power In-Sensor Neuronal Computing with Oscillatory Retinal Neurons for Frequency-Multiplexed, Parallel Machine Vision

Ultra-low Power In-Sensor Neuronal Computing with Oscillatory Retinal Neurons for Frequency-Multiplexed, Parallel Machine Vision

Multimodal In-Sensor Computing Implemented by Easily-Fabricated Oxide-Heterojunction Optoelectronic Synapses

Nanomaterial-Based Artificial Vision Systems: From Bioinspired Electronic Eyes to In-Sensor Processing Devices

Research and Application of Visual Object Recognition System Based on Deep Learning and Neural Morphological Computation

Broadband sensory networks with locally stored responsivities for neuromorphic machine vision

Bioinspired and Low-Power 2D Machine Vision with Adaptive Machine Learning and Forgetting

Intelligent Optoelectronic Devices for Next‐Generation Artificial Machine Vision

A 4.57 Μw@120fps Vision System of Sensing with Computing for BNN-Based Perception Applications

Solving Vision Tasks with Simple Photoreceptors Instead of Cameras