Abstract:Humans naturally perform audiovisual speech recognition (AVSR), enhancing the accuracy and robustness by integrating auditory and visual information. Spiking neural networks (SNNs), which mimic the brain's information-processing mechanisms, are well-suited for emulating the human capability of AVSR. Despite their potential, research on SNNs for AVSR is scarce, with most existing audio-visual multimodal methods focused on object or digit recognition. These models simply integrate features from both modalities, neglecting their unique characteristics and interactions. Additionally, they often rely on future information for current processing, which increases recognition latency and limits real-time applicability. Inspired by human speech perception, this paper proposes a novel human-inspired SNN named HI-AVSNN for AVSR, incorporating three key characteristics: cueing interaction, causal processing and spike activity. For cueing interaction, we propose a visual-cued auditory attention module (VCA2M) that leverages visual cues to guide attention to auditory features. We achieve causal processing by aligning the SNN's temporal dimension with that of visual and auditory features and applying temporal masking to utilize only past and current information. To implement spike activity, in addition to using SNNs, we leverage the event camera to capture lip movement as spikes, mimicking the human retina and providing efficient visual data. We evaluate HI-AVSNN on an audiovisual speech recognition dataset combining the DVS-Lip dataset with its corresponding audio samples. Experimental results demonstrate the superiority of our proposed fusion method, outperforming existing audio-visual SNN fusion methods and achieving a 2.27% improvement in accuracy over the only existing SNN-based AVSR method.

An Experimental Analysis of Different Approaches to Audio–Visual Speech Recognition and Lip-Reading

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition

AudioVSR: Enhancing Video Speech Recognition with Audio Data

Audio-Visual Speech Enhancement with Deep Multi-modality Fusion

Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices

MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition

Auxiliary Multimodal LSTM for Audio-visual Speech Recognition and Lipreading

For end-to-end audio-visual speech recognition

Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition

Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder

Unified Cross-Modal Attention: Robust Audio-Visual Speech Recognition and Beyond

Audio Visual Speech Recognition using Deep Recurrent Neural Networks

Modality Attention for End-to-end Audio-visual Speech Recognition.

An Investigation into Audio–Visual Speech Recognition under a Realistic Home–TV Scenario

Large-vocabulary Audio-visual Speech Recognition in Noisy Environments

Deep Multimodal Learning for Audio-Visual Speech Recognition

Robust Audio-Visual Speech Recognition Based on Hybrid Fusion

Robust end-to-end deep audiovisual speech recognition

Human-Inspired Audio-Visual Speech Recognition: Spike Activity, Cueing Interaction and Causal Processing

Audio-visual Recognition of Overlapped speech for the LRS2 dataset