Abstract:The advent of big data and the Internet of Things has created urgent demands for in-sensor computing hardware with multimodal perception that can effectively resolve the inefficiency, high latency, and excessive energy consumption challenges faced by conventional sensory systems. Here, a simple-structured optoelectronic synaptic device of In2O3SnO2/Nb:SrTiO3 (ITO/NSTO) heterostructure is proposed, which vividly demonstrates in-sensor computing and multimodal perception capabilities. First, with the ingenious synaptic responses of the device under both optical and electrical stimuli, a multimodal in-sensor neuromorphic computing system capable of concurrently perceiving and processing visual and auditory information is constructed. Using this multimodal system to perform a human emotion recognition task, the misjudgment arising from single-modal cognition can thereby be effectively avoided. Second, utilizing the integrated sensing and processing functions, along with the dynamic memories of the device array, a neuromorphic vision system is implemented for real-time monitoring of moving vehicles, which displays high recognition efficiency and accuracy. This research not only provides an optoelectronic synaptic device that is low-cost and easy to mass-produce, but also paves the way for next-generation in-sensor computing that can efficiently process multimodal signals. The separated sensing, storing, and computing modules along with single-mode perception in the conventional sensory system cause time-delay, massive energy consumption, as well as incomprehensive cognition issues. Here a neuromorphic device capable of performing multimodal perception and in-sensor computation within a single device is proposed, enabling a novel multimodal sensory system that is efficient, low-energy, and has minimal time latency. image

Advances in silicon-based in-sensor computing for neuromorphic vision sensors

Recent advances in in-sensor computational vision sensors: from mechanisms to applications

Artificial intelligence-based spatio-temporal vision sensors: applications and prospects

Low-Dimensional Optoelectronic Synaptic Devices for Neuromorphic Vision Sensors

Recent Advances in Imaging Devices: Image Sensors and Neuromorphic Vision Sensors

Advances in Neuromorphic Computing: Expanding Horizons for AI Development through Novel Artificial Neurons and In-Sensor Computing

Multimodal In-Sensor Computing Implemented by Easily-Fabricated Oxide-Heterojunction Optoelectronic Synapses

Bioinspired sensing-memory-computing integrated vision systems: biomimetic mechanisms, design principles, and applications

Event-Based Neuromorphic Vision for Autonomous Driving: A Paradigm Shift for Bio-Inspired Visual Sensing and Perception.

Mechanisms and Applications of Neuromorphic Sensors for Intelligent Visual Perception

Progress of Materials and Devices for Neuromorphic Vision Sensors

Processing Near Sensor Architecture in Mixed-Signal Domain with CMOS Image Sensor of Convolutional-Kernel-Readout Method

A robust neuromorphic vision sensor with optical control of ferroelectric switching

Optoelectronic Devices for In-Sensor Computing

Retinomorphic optoelectronic devices for intelligent machine vision.

Broadband sensory networks with locally stored responsivities for neuromorphic machine vision

Emerging photoelectric devices for neuromorphic vision applications: principles, developments, and outlooks

Photogated Synaptic Transistors Based on the Heterostructure of 4H-Sic and Organic Semiconductors for Neuromorphic Ultraviolet Vision

In-sensor computing for machine vision

Computational event-driven vision sensors for in-sensor spiking neural networks

A Three-Dimensional Neuromorphic Photosensor Array for Nonvolatile In-Sensor Computing