Abstract:Serving as the "eyes" and "ears" of the Internet of Things, optical and acoustic sensors are the fundamental components in hardware systems. Nowadays, mainstream hardware systems, often comprising numerous discrete sensors, conversion modules, and processing units, tend to result in complex architectures that are less efficient compared to human sensory pathways. Here, a visual-audio photodetector inspired by the human perception system is proposed to enable all-in-one visual and acoustic signal detection with computing capability. This device not only captures light but also optically records sound waves, thus achieving "watching" and "listening" within a single unit. The gate-tunable positive, negative, and zero photoresponses lead to highly programmable responsivities. This programmability enables the execution of diverse functions, including visual feature extraction, object classification, and sound wave manipulation. These results showcase the potential of expanding perception approaches in neuromorphic devices, opening up new possibilities to craft intelligent and compact hardware systems.

What problem does this paper attempt to address?

The main problems that this paper attempts to solve are the complexity and inefficiency in current optical and acoustic sensor systems. Specifically, mainstream hardware systems are usually composed of multiple discrete sensors, conversion modules and processing units, which leads to a complex architecture and low efficiency, especially when compared with the human perception path. In addition, traditional detection and processing modules are physically separated, which makes the data conversion, transmission and storage processes burdensome and further affects the operating efficiency of the processing modules. To solve these problems, the author proposes a bio - inspired visual - audio photodetector (VAPD) inspired by the human perception system, which can achieve the detection and computing capabilities of visual and audio signals within a single unit. This device can not only capture light, but also record sound waves through light, thus achieving the functions of "seeing" and "listening". Through gate - tunable positive, negative and zero photoelectric responses, the device has a highly programmable sensitivity and can perform multiple functions, including visual feature extraction, object classification and sound wave regulation. The following are the key points mentioned in the paper: 1. **Limitations of traditional systems**: - Current sensor systems are usually composed of multiple discrete sensors, conversion modules and processing units, resulting in a complex architecture and low efficiency. - Traditional detection and processing modules are physically separated, increasing the burden of data conversion, transmission and storage and affecting the operating efficiency of the processing modules. 2. **Design of bio - inspired visual - audio photodetector**: - This device is based on vertically stacked graphene - germanium (Gra - Ge) hybrid field - effect phototransistors. By controlling the interaction between the source - drain and gate leakage currents through the gate voltage, it realizes continuously tunable positive photocurrent (PPC) and negative photocurrent (NPC). - The device has remarkable responsiveness and operating speed and can capture the light source changes caused by sound waves. - Through gate - voltage programming, the device can not only pre - process visual images and perform target recognition tasks, but also adjust the sound wave intensity. 3. **Innovations**: - The device can simultaneously detect and compute visual and audio signals within a single unit, greatly simplifying the hardware design. - It presents new possibilities for developing intelligent and compact sensor systems, especially in fields such as the Internet of Things, security monitoring, smart homes, unmanned driving and human - machine interaction. In summary, this paper aims to solve the complexity and inefficiency problems in existing sensor systems by designing a bio - inspired visual - audio photodetector, thereby achieving a more efficient and intelligent hardware system.

Bionic visual-audio photodetectors with in-sensor perception and preprocessing

Multimodal In-Sensor Computing Implemented by Easily-Fabricated Oxide-Heterojunction Optoelectronic Synapses

Mechanisms and Applications of Neuromorphic Sensors for Intelligent Visual Perception

Bionic Visual‐Auditory Perceptual System Based on Ionotronic Neuromorphic Transistor for Information Encryption and Decryption with Sound Recognition Functions

Advances in silicon-based in-sensor computing for neuromorphic vision sensors

Birdlike broadband neuromorphic visual sensor arrays for fusion imaging

Integrated In-Sensor Computing Optoelectronic Device for Environment-Adaptable Artificial Retina Perception Application.

Progress in Bioinspired Photodetectors Design for Visual Information Processing

Recent Advances in Transistor-Based Bionic Perceptual Devices for Artificial Sensory Systems

An Optoelectronically Modulated Artificial Bionic Visual Neuron

IGZO/InHfO$_{\textit{x}}$ Nanowires/IGZO Phototransistor with Persistent Photoconductivity Effect for Intelligent Visual Perception Application

Neuromorphic sensory computing

Self-powered and broadband opto-sensor with bionic visual adaptation function based on multilayer γ-InSe flakes

Emerging photoelectric devices for neuromorphic vision applications: principles, developments, and outlooks

A 4.57 Μw@120fps Vision System of Sensing with Computing for BNN-Based Perception Applications

Neuromorphic Devices for Bionic Sensing and Perception

Adaptative machine vision with microsecond-level accurate perception beyond human retina

A Heterosynapse-Inspired Photodetector for Spatiotemporal Feature Fusion

Artificial Multisensory System with Optical Feedback for Multimodal Perceptual Imaging

Artificial multisensory integration nervous system with haptic and iconic perception behaviors

A Dual-Functional Integration of Photodetectors and Artificial Optoelectronic Synapses on a VO2/WO3 Heterojunction Device