Abstract:Due to the detrimental impact of noise on the conventional audio speech recognition (ASR) task, audio-visual speech recognition~(AVSR) has been proposed by incorporating both audio and visual video signals. Although existing methods have demonstrated that the aligned visual input of lip movements can enhance the robustness of AVSR systems against noise, the paired videos are not always available during inference, leading to the problem of the missing visual modality, which restricts their practicality in real-world scenarios. To tackle this problem, we propose a Discrete Feature based Visual Generative Model (DFVGM) which exploits semantic correspondences between the audio and visual modalities during training, generating visual hallucinations in lieu of real videos during inference. To achieve that, the primary challenge is to generate the visual hallucination given the noisy audio while preserving semantic correspondences with the clean speech. To tackle this challenge, we start with training the audio encoder in the Audio-Only (AO) setting, which generates continuous semantic features closely associated with the linguistic information. Simultaneously, the visual encoder is trained in the Visual-Only (VO) setting, producing visual features that are phonetically related. Next, we employ K-means to discretize the continuous audio and visual feature spaces. The discretization step allows DFVGM to capture high-level semantic structures that are more resilient to noise and generate visual hallucinations with high quality. To evaluate the effectiveness and robustness of our approach, we conduct extensive experiments on two publicly available datasets. The results demonstrate that our method achieves a remarkable 53% relative reduction (30.5%->12.9%) in Word Error Rate (WER) on average compared to the current state-of-the-art Audio-Only (AO) baselines while maintaining comparable results (< 5% difference) under the Audio-Visual (AV) setting even without video as input.

Tri-Modal Speech Recognition for Noisy and Variable Lighting Conditions

Robust tri-modal automatic speech recognition for consumer applications.

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

An Investigation into Audio–Visual Speech Recognition under a Realistic Home–TV Scenario

Robust Audio-visual Speech Recognition Using Bimodal Dfsmn with Multi-condition Training and Dropout Regularization.

For end-to-end audio-visual speech recognition

How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition

Multimodal Speech Recognition Using EEG and Audio Signals: A Novel Approach for Enhancing ASR Systems

Modality Attention for End-to-end Audio-visual Speech Recognition.

DCIM-AVSR : Efficient Audio-Visual Speech Recognition via Dual Conformer Interaction Module

Audio-visual multi-channel speech separation, dereverberation and recognition

Multisensory benefits for speech recognition in noisy environments

Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer

Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition

AVATAR: Unconstrained Audiovisual Speech Recognition

Robust end-to-end deep audiovisual speech recognition

Multimodal audiovisual speech recognition architecture using a three‐feature multi‐fusion method for noise‐robust systems

Visual Hallucination Elevates Speech Recognition

Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition

AV Taris: Online Audio-Visual Speech Recognition

Unified Cross-Modal Attention: Robust Audio-Visual Speech Recognition and Beyond