Abstract:Abstract Human behaviors are extremely sophisticated, relying on the adaptive, plastic and event-driven network of sensory neurons. Such neuronal system analyzes multiple sensory cues efficiently to establish accurate depiction of the environment. Here, we develop a bimodal artificial sensory neuron to implement the sensory fusion processes. Such a bimodal artificial sensory neuron collects optic and pressure information from the photodetector and pressure sensors respectively, transmits the bimodal information through an ionic cable, and integrates them into post-synaptic currents by a synaptic transistor. The sensory neuron can be excited in multiple levels by synchronizing the two sensory cues, which enables the manipulating of skeletal myotubes and a robotic hand. Furthermore, enhanced recognition capability achieved on fused visual/haptic cues is confirmed by simulation of a multi-transparency pattern recognition task. Our biomimetic design has the potential to advance technologies in cyborg and neuromorphic systems by endowing them with supramodal perceptual capabilities.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **How to develop an artificial sensory neuron capable of integrating visual and tactile information to improve the reliability and accuracy of machine perception systems?** Specifically, the research aims to develop a bimodal artificial sensory neuron (BASE) by imitating the multi - modal perception fusion mechanism in the biological nervous system, thereby achieving a more accurate description of the environment and endowing it with super - modal perception capabilities in bionic and neuromorphic systems. ### Specific background of the problem 1. **Advantages of biological systems**: - When interacting with the dynamic real world, biological systems exhibit higher complexity and adaptability than electronic systems. This is because biological systems rely on self - adaptive, plastic, and event - driven sensory neural networks and can efficiently analyze multiple sensory cues. 2. **Limitations of existing technologies**: - Although some bio - inspired systems have achieved complex sensorimotor functions using silicon - based circuits and software, the efficiency of these systems is still affected when the amount of data increases because they rely on centralized and sequential operations. - Current technologies lack a platform to process multi - modal sensory data, resulting in the inability to achieve super - modal perception capabilities, that is, using information from multiple sensory channels simultaneously to improve the accuracy and reliability of perception. ### Core contributions of the paper 1. **Development of the bimodal artificial sensory neuron (BASE)**: - BASE consists of four core components: a resistive pressure sensor, a perovskite - based photodetector, a hydrogel - based ionic cable, and a synaptic transistor. These components are respectively responsible for receiving and converting external visual and tactile stimuli into electrical signals and transmitting them through the ionic cable to the synaptic transistor for integration and conversion into transient channel currents (similar to the biological excitatory postsynaptic current EPSC). 2. **Application of multi - modal perception fusion**: - The researchers demonstrated that BASE can be used to control skeletal muscle cells and robotic hands, simulating the "perception - action" process of organisms. - By designing and simulating a BASE matrix as the feature extraction layer of the perceptron, the researchers verified that multi - modal perception fusion can improve the recognition rate of multi - transparency letter patterns and maintain high recognition accuracy even when the amount of data is reduced. ### Conclusion This research has developed a new type of artificial sensory neuron by imitating the multi - modal perception fusion mechanism in the biological nervous system, which can achieve super - modal perception capabilities in bionic and neuromorphic systems, thereby significantly improving the reliability and accuracy of machine perception systems. This achievement is of great significance for the future development of neuro - robots, cyborg systems, and autonomous artificial intelligence.

An artificial sensory neuron with visual-haptic fusion

An Artificial Sensory Neuron with Tactile Perceptual Learning

Artificial Multisensory Neurons with Fused Haptic and Temperature Perception for Multimodal In‐Sensor Computing

Bimodal Bioinspired Sensory Neuron With Tactile and Visual Perceptual Capabilities

Artificial multisensory integration nervous system with haptic and iconic perception behaviors

Artificial Multisensory Neuron with a Single Transistor for Multimodal Perception through Hybrid Visual and Thermal Sensing

Artificial visual‐tactile perception array for enhanced memory and neuromorphic computations

Multimodal Artificial Neurological Sensory-Memory System Based on Flexible Carbon Nanotube Synaptic Transistor

An artificial neuromorphic somatosensory system with spatio-temporal tactile perception and feedback functions

Visualized in-sensor computing

A Heterogeneously Integrated Spiking Neuron Array for Multimode‐Fused Perception and Object Classification

Multimodal In-Sensor Computing Implemented by Easily-Fabricated Oxide-Heterojunction Optoelectronic Synapses

Artificial organic afferent nerves enable closed-loop tactile feedback for intelligent robot

A bioinspired analogous nerve towards artificial intelligence

Bioinspired multisensory neural network with crossmodal integration and recognition

A Spiking Visual Neuron for Depth Perceptual Systems

Mammalian-brain-inspired neuromorphic motion-cognition nerve achieves cross-modal perceptual enhancement

A Flexible Artificial Sensory Nerve Enabled by Nanoparticle‐Assembled Synaptic Devices for Neuromorphic Tactile Recognition (Adv. Sci. 24/2022)

An Artificial Spiking Afferent Nerve Based on Synaptic Transistor for Thermal Perception

An evaluation of recent advancements in biological sensory organ-inspired neuromorphically tuned biomimetic devices

An Artificial Visual Neuron with Multiplexed Rate and Time-to-first-spike Coding