Abstract:The current accuracy of speech recognition can reach over 97% on different datasets, but in noisy environments, it is greatly reduced. Improving speech recognition performance in noisy environments is a challenging task. Due to the fact that visual information is not affected by noise, researchers often use lip information to help to improve speech recognition performance. This is where the performance of lip recognition and the effect of cross-modal fusion are particularly important. In this paper, we try to improve the accuracy of speech recognition in noisy environments by improving the lip reading performance and the cross-modal fusion effect. First, due to the same lip possibly containing multiple meanings, we constructed a one-to-many mapping relationship model between lips and speech allowing for the lip reading model to consider which articulations are represented from the input lip movements. Audio representations are also preserved by modeling the inter-relationships between paired audiovisual representations. At the inference stage, the preserved audio representations could be extracted from memory by the learned inter-relationships using only video input. Second, a joint cross-fusion model using the attention mechanism could effectively exploit complementary intermodal relationships, and the model calculates cross-attention weights on the basis of the correlations between joint feature representations and individual modalities. Lastly, our proposed model achieved a 4.0% reduction in WER in a -15 dB SNR environment compared to the baseline method, and a 10.1% reduction in WER compared to speech recognition. The experimental results show that our method could achieve a significant improvement over speech recognition models in different noise environments.

Regression Based Landmark Estimation and Multi-Feature Fusion for Visual Speech Recognition.

AudioVSR: Enhancing Video Speech Recognition with Audio Data

LipFormer: Learning to Lipread Unseen Speakers based on Visual-Landmark Transformers

Audio-Visual Speech Recognition Using A Two-Step Feature Fusion Strategy.

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

Landmark-Guided Cross-Speaker Lip Reading with Mutual Information Regularization

Robust Audio-Visual Speech Recognition Based on Hybrid Fusion

Visual Features Extracting & Selecting For Lipreading

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

Lip Graph Assisted Audio-Visual Speech Recognition Using Bidirectional Synchronous Fusion.

A Novel Lip Descriptor for Audio-Visual Keyword Spotting Based on Adaptive Decision Fusion

Visual Information Assisted Mandarin Large Vocabulary Continuous Speech Recognition

Speech Guided Disentangled Visual Representation Learning for Lip Reading

Spatio-Temporal Fusion Based Convolutional Sequence Learning for Lip Reading

Audio-Visual System for Robust Speaker Recognition.

Improving Speech Recognition Performance in Noisy Environments by Enhancing Lip Reading Accuracy

Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers

Fusing information streams in end-to-end audio-visual speech recognition

Mmmic: Multi-modal Speech Recognition Based on Mmwave Radar.

Generalizing sentence-level lipreading to unseen speakers: a two-stream end-to-end approach

Lipreading Architecture Based on Multiple Convolutional Neural Networks for Sentence-Level Visual Speech Recognition