Abstract:Audio-Visual Speech Recognition (AVSR) is a promising approach to improving the accuracy and robustness of speech recognition systems with the assistance of visual cues in challenging acoustic environments. In this paper, we present a novel audio-visual speech recognition architecture with unified cross-modal attention. Our approach concatenates the sequences temporally from different modalities and encodes the fused sequence in the unified feature space using a shared Conformer encoder. We then explicitly model additive noise and potential out-of-sync samples during training, and propose an auxiliary asynchronization-aware loss to improve the system performance on out-of-sync data. To enhance the efficacy of unified cross-modal attention, a manual attention alignment strategy is designed and applied to the model, bringing additional gains in both recognition accuracy and computation cost. As demonstrated by experiments on the large-scale audio-visual LRS3 dataset, our proposed approach reduces the word error rate (WER) by relatively 50 compared to the audio-only single-modal ASR system under noisy conditions, and relatively 25 compared to the previous audio-visual ASR baseline. The proposed audio-visual ASR system also shows superior robustness in more challenging conditions, such as audio-only data, visual corruption, audio-visual misalignment, and multi-talker interference. Moreover, the proposed Unified Cross-Modal Attention model exhibits a more general ability in multi-modality fusion, allowing for easy integration of additional modalities into the model with this framework to achieve a more accurate, robust, and safer multi-modal system.

Robust Audio-visual Speech Recognition Using Bimodal Dfsmn with Multi-condition Training and Dropout Regularization.

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

Audio-Visual Speech Enhancement with Deep Multi-modality Fusion

AudioVSR: Enhancing Video Speech Recognition with Audio Data

An Investigation into Audio–Visual Speech Recognition under a Realistic Home–TV Scenario

Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder

Robust Audio-Visual Speech Recognition Based on Hybrid Fusion

Robust Audio-Visual Mandarin Speech Recognition Based on Adaptive Decision Fusion and Tone Features

Auxiliary Multimodal LSTM for Audio-visual Speech Recognition and Lipreading

DCIM-AVSR : Efficient Audio-Visual Speech Recognition via Dual Conformer Interaction Module

A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition

How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition

Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer

MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition

Robust end-to-end deep audiovisual speech recognition

Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition

Multimodal audiovisual speech recognition architecture using a three‐feature multi‐fusion method for noise‐robust systems

For end-to-end audio-visual speech recognition

Unified Cross-Modal Attention: Robust Audio-Visual Speech Recognition and Beyond

Reliability-Based Large-Vocabulary Audio-Visual Speech Recognition

Improving Audio-visual Speech Recognition Performance with Cross-modal Student-teacher Training