Abstract:The advent of big data and the Internet of Things has created urgent demands for in-sensor computing hardware with multimodal perception that can effectively resolve the inefficiency, high latency, and excessive energy consumption challenges faced by conventional sensory systems. Here, a simple-structured optoelectronic synaptic device of In2O3SnO2/Nb:SrTiO3 (ITO/NSTO) heterostructure is proposed, which vividly demonstrates in-sensor computing and multimodal perception capabilities. First, with the ingenious synaptic responses of the device under both optical and electrical stimuli, a multimodal in-sensor neuromorphic computing system capable of concurrently perceiving and processing visual and auditory information is constructed. Using this multimodal system to perform a human emotion recognition task, the misjudgment arising from single-modal cognition can thereby be effectively avoided. Second, utilizing the integrated sensing and processing functions, along with the dynamic memories of the device array, a neuromorphic vision system is implemented for real-time monitoring of moving vehicles, which displays high recognition efficiency and accuracy. This research not only provides an optoelectronic synaptic device that is low-cost and easy to mass-produce, but also paves the way for next-generation in-sensor computing that can efficiently process multimodal signals. The separated sensing, storing, and computing modules along with single-mode perception in the conventional sensory system cause time-delay, massive energy consumption, as well as incomprehensive cognition issues. Here a neuromorphic device capable of performing multimodal perception and in-sensor computation within a single device is proposed, enabling a novel multimodal sensory system that is efficient, low-energy, and has minimal time latency. image

Artificial Multisensory System with Optical Feedback for Multimodal Perceptual Imaging

Multimodal In-Sensor Computing Implemented by Easily-Fabricated Oxide-Heterojunction Optoelectronic Synapses

A Multimodal Sensing CMOS Imager Based on Dual‐Focus Imaging(Adv. Sci. 14/2023)

Artificial multisensory integration nervous system with haptic and iconic perception behaviors

Artificial Multisensory Neurons with Fused Haptic and Temperature Perception for Multimodal In‐Sensor Computing

Artificial Tactile Sensing System with Photoelectric Output for High Accuracy Haptic Texture Recognition and Parallel Information Processing

A sensory memory processing system with multi-wavelength synaptic-polychromatic light emission for multi-modal information recognition

Simulating Tactile and Visual Multisensory Behaviour in Humans Based on an MoS2 Field Effect Transistor

An artificial neuromorphic somatosensory system with spatio-temporal tactile perception and feedback functions

Artificial visual‐tactile perception array for enhanced memory and neuromorphic computations

Artificial Multisensory Neuron with a Single Transistor for Multimodal Perception through Hybrid Visual and Thermal Sensing

Multimodal Artificial Neurological Sensory-Memory System Based on Flexible Carbon Nanotube Synaptic Transistor

Optical skin: Sensor-integration-free multimodal flexible sensing

A Universal Flexible Near-sensor Neuromorphic Tactile System with Multi-threshold strategy for Pressure Characteristic Detection

Bio‐Inspired Artificial Perceptual Devices for Neuromorphic Computing and Gesture Recognition

A Self-Powered Brain Multi-Perception Receptor for Sensory-Substitution Application

A Fully Integrated Sensor-Brain-Machine Interface System for Restoring Somatosensation

Integrated In-Sensor Computing Optoelectronic Device for Environment-Adaptable Artificial Retina Perception Application.

An artificial sensory neuron with visual-haptic fusion

Bioinspired multisensory neural network with crossmodal integration and recognition

Multi-modal Sensory Feedback System for Upper Limb Amputees