Abstract:The fusion of audio and visual modalities is an important stage of audio-visual speech recognition (AVSR), which is generally approached through feature fusion or decision fusion. Feature fusion can exploit the covariations between features from different modalities effectively, whereas decision fusion shows the robustness of capturing an optimal combination of multimodality. In this work, to take full advantage of the complementarity of the two fusion strategies and address the challenge of inherent ambiguity in noisy environments, we propose a novel hybrid fusion based AVSR method with residual networks and Bidirectional Gated Recurrent Unit (BGRU), which is able to distinguish homophones in both clean and noisy conditions. Specifically, a simple yet effective audio-visual encoder is used to map audio and visual features into a shared latent space to capture more discriminative multi-modal feature and find the internal correlation between spatial-temporal information for different modalities. Furthermore, a decision fusion module is designed to get final predictions in order to robustly utilize the reliability measures of audio-visual information. Finally, we introduce a combined loss, which shows its noise-robustness in learning the joint representation across various modalities. Experimental results on the largest publicly available dataset (LRW) demonstrate the robustness of the proposed method under various noisy conditions.

Robust Multi-Stream Speech Recognition Based on Weighting the Output Probabilities of Feature Components

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

Combining Information from Multi-Stream Features Using Deep Neural Network in Speech Recognition

Noise Robust Speech Recognition Using Multi-Channel Based Channel Selection And ChannelWeighting.

Multi-Encoder Learning and Stream Fusion for Transformer-Based End-to-End Automatic Speech Recognition

Robust Audio-Visual Speech Recognition Based on Hybrid Fusion

Qifusion-Net: Layer-adapted Stream/Non-stream Model for End-to-End Multi-Accent Speech Recognition

Multi-Channel Feature Adaptation for Robust Speech Recognition

Robust Analysis And Weighting On Mfcc Components For Speech Recognition And Speaker Identification

speech and noise dual-stream spectrogram refine network with speech distortion loss for robust speech recognition

Weight Estimation for Audio-Visual Multi-level Fusion in Bimodal Speaker Identification

An information fusion framework with multi-channel feature concatenation and multi-perspective system combination for the deep-learning-based robust recognition of microphone array speech

Language model fusion for streaming end to end speech recognition

Stream Weight Training Based on MCE for Audio-Visual LVCSR

Steganalysis of AMR Speech Stream Based on Multi-Domain Information Fusion

Recognition of Speech under G -Force Based on Weighted Features of MFCC

Dbn Based Multi-Stream Models For Speech

Coarse-Grained Attention Fusion with Joint Training Framework for Complex Speech Enhancement and End-to-End Speech Recognition

Foster Strengths and Circumvent Weaknesses: a Speech Enhancement Framework with Two-branch Collaborative Learning

Audio-Visual Speech Recognition Using A Two-Step Feature Fusion Strategy.