Abstract:The advent of big data and the Internet of Things has created urgent demands for in-sensor computing hardware with multimodal perception that can effectively resolve the inefficiency, high latency, and excessive energy consumption challenges faced by conventional sensory systems. Here, a simple-structured optoelectronic synaptic device of In2O3SnO2/Nb:SrTiO3 (ITO/NSTO) heterostructure is proposed, which vividly demonstrates in-sensor computing and multimodal perception capabilities. First, with the ingenious synaptic responses of the device under both optical and electrical stimuli, a multimodal in-sensor neuromorphic computing system capable of concurrently perceiving and processing visual and auditory information is constructed. Using this multimodal system to perform a human emotion recognition task, the misjudgment arising from single-modal cognition can thereby be effectively avoided. Second, utilizing the integrated sensing and processing functions, along with the dynamic memories of the device array, a neuromorphic vision system is implemented for real-time monitoring of moving vehicles, which displays high recognition efficiency and accuracy. This research not only provides an optoelectronic synaptic device that is low-cost and easy to mass-produce, but also paves the way for next-generation in-sensor computing that can efficiently process multimodal signals. The separated sensing, storing, and computing modules along with single-mode perception in the conventional sensory system cause time-delay, massive energy consumption, as well as incomprehensive cognition issues. Here a neuromorphic device capable of performing multimodal perception and in-sensor computation within a single device is proposed, enabling a novel multimodal sensory system that is efficient, low-energy, and has minimal time latency. image

Crossmodal sensory neurons based on high-performance flexible memristors for human-machine in-sensor computing system

Multimodal In-Sensor Computing Implemented by Easily-Fabricated Oxide-Heterojunction Optoelectronic Synapses

A Bioinspired MXene-based Flexible Sensory Neuron for Tactile Near-Sensor Computing

A Brain-Inspired In-Memory Computing System for Neuronal Communication Via Memristive Circuits.

Artificial Multisensory Neurons with Fused Haptic and Temperature Perception for Multimodal In‐Sensor Computing

Multimode Modulated Memristors for In-Sensor Computing System

A Low-Power and Flexible Bioinspired Artificial Sensory Neuron Capable of Tactile Perceptual and Associative Learning

Multimodal Neuromorphic Sensory-Processing System With Memristor Circuits for Smart Home Applications

A calibratable sensory neuron based on epitaxial VO2 for spike-based neuromorphic multisensory system

A Heterogeneously Integrated Spiking Neuron Array for Multimode‐Fused Perception and Object Classification

Tunable flexible artificial synapses: a new path toward a wearable electronic system

Stretchable, Skin-Conformable Neuromorphic System for Tactile Sensory Recognizing and Encoding

MXene‐ZnO Memristor for Multimodal In‐Sensor Computing

A neuromorphic physiological signal processing system based on VO2 memristor for next-generation human-machine interface

An artificial sensory neuron with visual-haptic fusion

Multimodal Artificial Neurological Sensory-Memory System Based on Flexible Carbon Nanotube Synaptic Transistor

An Artificial Spiking Afferent Nerve Based on Mott Memristors for Neurorobotics

Energy-efficient Flexible Photoelectric Device with 2D/0D Hybrid Structure for Bio-Inspired Artificial Heterosynapse Application

Bioinspired Artificial Sensory Nerve Based on Nafion Memristor

A Dual-Mode Organic Memristor for Coordinated Visual Perceptive Computing

Multifunctional Optoelectronic Memristor Based on CeO2/MoS2 Heterojunction for Advanced Artificial Synapses and Bionic Visual System with Nociceptive Sensing