Abstract:The current accuracy of speech recognition can reach over 97% on different datasets, but in noisy environments, it is greatly reduced. Improving speech recognition performance in noisy environments is a challenging task. Due to the fact that visual information is not affected by noise, researchers often use lip information to help to improve speech recognition performance. This is where the performance of lip recognition and the effect of cross-modal fusion are particularly important. In this paper, we try to improve the accuracy of speech recognition in noisy environments by improving the lip reading performance and the cross-modal fusion effect. First, due to the same lip possibly containing multiple meanings, we constructed a one-to-many mapping relationship model between lips and speech allowing for the lip reading model to consider which articulations are represented from the input lip movements. Audio representations are also preserved by modeling the inter-relationships between paired audiovisual representations. At the inference stage, the preserved audio representations could be extracted from memory by the learned inter-relationships using only video input. Second, a joint cross-fusion model using the attention mechanism could effectively exploit complementary intermodal relationships, and the model calculates cross-attention weights on the basis of the correlations between joint feature representations and individual modalities. Lastly, our proposed model achieved a 4.0% reduction in WER in a -15 dB SNR environment compared to the baseline method, and a 10.1% reduction in WER compared to speech recognition. The experimental results show that our method could achieve a significant improvement over speech recognition models in different noise environments.

Improved Speech Feature Extraction and Its Application

Speech Emotion Recognition Based on Syllable-Level Feature Extraction

Real-time Speech Emotion Recognition Based on Syllable-Level Feature Extraction

Noise Estimation Using Mean Square Cross Prediction Error for Speech Enhancement

Speech Personality Recognition Based on Annotation Classification Using Log-Likelihood Distance and Extraction of Essential Audio Features.

Wavoice: A mmWave-assisted Noise-resistant Speech Recognition SystemJust Accepted

Further Feature Extraction and Its Application on Speaker Recognition

Auditory Model Based Speech Feature Extraction and Its Application to Speaker Identification

Feature Selection In Mandarin Large Vocabulary Continuous Speech Recognition

Feature Extraction in Speechreading.

Improved speech recognition algorithm based on MFCC feature

Improvement of Speech Detection Using ERB Feature Extraction

LDA Applied to Mandarin Speech Recognition System

Auditory model-based speech feature extraction and its application to speaker identification

Improving Speech Recognition Performance in Noisy Environments by Enhancing Lip Reading Accuracy

Robust Speech Recognition by Selecting Mel-Filter Banks

A Noise Robust Front End Algorithm for Mandarin Speech Recognition and Performance Analysis

Experiments and Analysis for Speaker Dependent Mandarin Syllable Recognition.

Research on Speech Recognition Methods Based on Spectral Subtraction and Improved BP Neural Network Algorithm

Design and implementation of speech recognition algorithm based on frequency range