Abstract:Audio-Visual Speech Recognition (AVSR) is a promising approach to improving the accuracy and robustness of speech recognition systems with the assistance of visual cues in challenging acoustic environments. In this paper, we present a novel audio-visual speech recognition architecture with unified cross-modal attention. Our approach concatenates the sequences temporally from different modalities and encodes the fused sequence in the unified feature space using a shared Conformer encoder. We then explicitly model additive noise and potential out-of-sync samples during training, and propose an auxiliary asynchronization-aware loss to improve the system performance on out-of-sync data. To enhance the efficacy of unified cross-modal attention, a manual attention alignment strategy is designed and applied to the model, bringing additional gains in both recognition accuracy and computation cost. As demonstrated by experiments on the large-scale audio-visual LRS3 dataset, our proposed approach reduces the word error rate (WER) by relatively 50 compared to the audio-only single-modal ASR system under noisy conditions, and relatively 25 compared to the previous audio-visual ASR baseline. The proposed audio-visual ASR system also shows superior robustness in more challenging conditions, such as audio-only data, visual corruption, audio-visual misalignment, and multi-talker interference. Moreover, the proposed Unified Cross-Modal Attention model exhibits a more general ability in multi-modality fusion, allowing for easy integration of additional modalities into the model with this framework to achieve a more accurate, robust, and safer multi-modal system.

SIR-Progressive Audio-Visual TF-Gridnet with ASR-Aware Selector for Target Speaker Extraction in MISP 2023 Challenge

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

Audio-Visual Speech Enhancement with Deep Multi-modality Fusion

The Multimodal Information Based Speech Processing (MISP) 2023 Challenge: Audio-Visual Target Speaker Extraction

Scenario-Aware Audio-Visual TF-GridNet for Target Speech Extraction

The NPU-ASLP System for Audio-Visual Speech Recognition in MISP 2022 Challenge

Summary on the Multimodal Information-Based Speech Processing (MISP) 2023 Challenge

MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition

Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition

Improving Visual Speech Enhancement Network by Learning Audio-visual Affinity with Multi-head Attention

Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder

KS-Net: Multi-band joint speech restoration and enhancement network for 2024 ICASSP SSI Challenge

Unified Cross-Modal Attention: Robust Audio-Visual Speech Recognition and Beyond

Summary on the Multimodal Information Based Speech Processing (MISP) 2022 Challenge.

Attentive Fusion Enhanced Audio-Visual Encoding for Transformer Based Robust Speech Recognition

Progressive Multi-Target Network Based Speech Enhancement with Snr-Preselection for Robust Speaker Diarization

CATNet: Cross-modal fusion for audio-visual speech recognition

An audio-quality-based multi-strategy approach for target speaker extraction in the MISP 2023 Challenge

Multi-Stage Progressive Speech Enhancement Network

MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition

An Investigation into Audio–Visual Speech Recognition under a Realistic Home–TV Scenario