Abstract:Due to the detrimental impact of noise on the conventional audio speech recognition (ASR) task, audio-visual speech recognition~(AVSR) has been proposed by incorporating both audio and visual video signals. Although existing methods have demonstrated that the aligned visual input of lip movements can enhance the robustness of AVSR systems against noise, the paired videos are not always available during inference, leading to the problem of the missing visual modality, which restricts their practicality in real-world scenarios. To tackle this problem, we propose a Discrete Feature based Visual Generative Model (DFVGM) which exploits semantic correspondences between the audio and visual modalities during training, generating visual hallucinations in lieu of real videos during inference. To achieve that, the primary challenge is to generate the visual hallucination given the noisy audio while preserving semantic correspondences with the clean speech. To tackle this challenge, we start with training the audio encoder in the Audio-Only (AO) setting, which generates continuous semantic features closely associated with the linguistic information. Simultaneously, the visual encoder is trained in the Visual-Only (VO) setting, producing visual features that are phonetically related. Next, we employ K-means to discretize the continuous audio and visual feature spaces. The discretization step allows DFVGM to capture high-level semantic structures that are more resilient to noise and generate visual hallucinations with high quality. To evaluate the effectiveness and robustness of our approach, we conduct extensive experiments on two publicly available datasets. The results demonstrate that our method achieves a remarkable 53% relative reduction (30.5%->12.9%) in Word Error Rate (WER) on average compared to the current state-of-the-art Audio-Only (AO) baselines while maintaining comparable results (< 5% difference) under the Audio-Visual (AV) setting even without video as input.

Incorporating Visual Information Reconstruction into Progressive Learning for Optimizing audio-visual Speech Enhancement

Audio-Visual Speech Enhancement with Deep Multi-modality Fusion

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

AudioVSR: Enhancing Video Speech Recognition with Audio Data

How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition

Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder

Audio-Visual Speech Separation in Noisy Environments with a Lightweight Iterative Model

Visual Hallucination Elevates Speech Recognition

Learning Contextually Fused Audio-visual Representations for Audio-visual Speech Recognition

Improved Lite Audio-Visual Speech Enhancement

Audio-Visual Class-Incremental Learning

Improving Visual Speech Enhancement Network by Learning Audio-visual Affinity with Multi-head Attention

An Attention Based Speaker-Independent Audio-Visual Deep Learning Model for Speech Enhancement

Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition

XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception

SNR-Based Progressive Learning of Deep Neural Network for Speech Enhancement

Speech inpainting: Context-based speech synthesis guided by video

LA-VocE: Low-SNR Audio-visual Speech Enhancement using Neural Vocoders

SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Model

Target Speech Extraction with Pre-trained AV-HuBERT and Mask-And-Recover Strategy

Cooperative Dual Attention for Audio-Visual Speech Enhancement with Facial Cues