Abstract:The advent of big data and the Internet of Things has created urgent demands for in-sensor computing hardware with multimodal perception that can effectively resolve the inefficiency, high latency, and excessive energy consumption challenges faced by conventional sensory systems. Here, a simple-structured optoelectronic synaptic device of In2O3SnO2/Nb:SrTiO3 (ITO/NSTO) heterostructure is proposed, which vividly demonstrates in-sensor computing and multimodal perception capabilities. First, with the ingenious synaptic responses of the device under both optical and electrical stimuli, a multimodal in-sensor neuromorphic computing system capable of concurrently perceiving and processing visual and auditory information is constructed. Using this multimodal system to perform a human emotion recognition task, the misjudgment arising from single-modal cognition can thereby be effectively avoided. Second, utilizing the integrated sensing and processing functions, along with the dynamic memories of the device array, a neuromorphic vision system is implemented for real-time monitoring of moving vehicles, which displays high recognition efficiency and accuracy. This research not only provides an optoelectronic synaptic device that is low-cost and easy to mass-produce, but also paves the way for next-generation in-sensor computing that can efficiently process multimodal signals. The separated sensing, storing, and computing modules along with single-mode perception in the conventional sensory system cause time-delay, massive energy consumption, as well as incomprehensive cognition issues. Here a neuromorphic device capable of performing multimodal perception and in-sensor computation within a single device is proposed, enabling a novel multimodal sensory system that is efficient, low-energy, and has minimal time latency. image

Bimodal Bioinspired Sensory Neuron With Tactile and Visual Perceptual Capabilities

An artificial sensory neuron with visual-haptic fusion

Artificial multisensory integration nervous system with haptic and iconic perception behaviors

Multimodal In-Sensor Computing Implemented by Easily-Fabricated Oxide-Heterojunction Optoelectronic Synapses

Artificial Multisensory Neuron with a Single Transistor for Multimodal Perception through Hybrid Visual and Thermal Sensing

Bioinspired multisensory neural network with crossmodal integration and recognition

Highly Promising 2D/1D BP‐C/CNT Bionic Opto‐Olfactory Co‐Sensory Artificial Synapses for Multisensory Integration

Artificial Multisensory Neurons with Fused Haptic and Temperature Perception for Multimodal In‐Sensor Computing

Bipolar synaptic organic/inorganic heterojunction transistor with complementary light modulation and low power consumption for energy-efficient artificial vision systems

Fabrication of carbon nanotube neuromorphic thin film transistor arrays and their applications for flexible olfactory-visual multisensory synergy recognition

Self-Powered Organic Optoelectronic Synapses with Binarized Weights for Noise-Suppressed Visual Perception and High-Robustness Inference

Tactile Sensory Coding and Learning with Bio-Inspired Optoelectronic Spiking Afferent Nerves

A Photosensitive Spiking Neuron Using a Single Band-Modulation Device with Tunable Spiking Sensitivity

Mammalian-brain-inspired neuromorphic motion-cognition nerve achieves cross-modal perceptual enhancement

Recent Advances in Transistor-Based Bionic Perceptual Devices for Artificial Sensory Systems

Simulating Tactile and Visual Multisensory Behaviour in Humans Based on an MoS2 Field Effect Transistor

All in One, Self‐Powered Bionic Artificial Nerve Based on a Triboelectric Nanogenerator

Bio-Inspired P-type TeSeOx Synaptic Transistor Based on Multispectral Sensing for Neuromorphic Visual Multilevel Nociceptor.

Oxide Synaptic Transistors Coupled with Triboelectric Nanogenerators for Bio-Inspired Tactile Sensing Application

A Heterogeneously Integrated Spiking Neuron Array for Multimode‐Fused Perception and Object Classification

An Artificial Sensory Neuron with Tactile Perceptual Learning