Abstract:Audio-visual speech recognition (AVSR) can effectively and significantly improve the recognition rates of small-vocabulary systems, compared to their audio-only counterparts. For large-vocabulary systems, however, there are still many difficulties, such as unsatisfactory video recognition accuracies, that make it hard to improve over audio-only baselines. In this paper, we specifically consider such scenarios, focusing on the large-vocabulary task of the LRS2 database, where audio-only performance is far superior to video-only accuracies, making this an interesting and challenging setup for multi-modal integration. To address the inherent difficulties, we propose a new fusion strategy: a recurrent integration network is trained to fuse the state posteriors of multiple single-modality models, guided by a set of model-based and signal-based stream reliability measures. During decoding, this network is used for stream integration within a hybrid recognizer, where it can thus cope with the time-variant reliability and information content of its multiple feature inputs. We compare the results with end-to-end AVSR systems as well as with competitive hybrid baseline models, finding that the new fusion strategy shows superior results, on average even outperforming oracle dynamic stream weighting, which has so far marked the -- realistically unachievable -- upper bound for standard stream weighting. Even though the pure lipreading performance is low, audio-visual integration is helpful under all -- clean, noisy, and reverberant -- conditions. On average, the new system achieves a relative word error rate reduction of 42.18\% compared to the audio-only model, pointing at a high effectiveness of the proposed integration approach.

Robust Audio-Visual ASR with Unified Cross-Modal Attention

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

Unified Cross-Modal Attention: Robust Audio-Visual Speech Recognition and Beyond

AudioVSR: Enhancing Video Speech Recognition with Audio Data

MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition

Modality Attention for End-to-end Audio-visual Speech Recognition.

Audio-Visual Speech Enhancement with Deep Multi-modality Fusion

For end-to-end audio-visual speech recognition

Robust Audio-Visual Speech Recognition Based on Hybrid Fusion

Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition

Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition

How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition

Large-vocabulary Audio-visual Speech Recognition in Noisy Environments

Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition

XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception

Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder

Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer

CATNet: Cross-modal fusion for audio-visual speech recognition

AlignVSR: Audio-Visual Cross-Modal Alignment for Visual Speech Recognition

Robust Audio-visual Speech Recognition Using Bimodal Dfsmn with Multi-condition Training and Dropout Regularization.

MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition