Abstract:The current accuracy of speech recognition can reach over 97% on different datasets, but in noisy environments, it is greatly reduced. Improving speech recognition performance in noisy environments is a challenging task. Due to the fact that visual information is not affected by noise, researchers often use lip information to help to improve speech recognition performance. This is where the performance of lip recognition and the effect of cross-modal fusion are particularly important. In this paper, we try to improve the accuracy of speech recognition in noisy environments by improving the lip reading performance and the cross-modal fusion effect. First, due to the same lip possibly containing multiple meanings, we constructed a one-to-many mapping relationship model between lips and speech allowing for the lip reading model to consider which articulations are represented from the input lip movements. Audio representations are also preserved by modeling the inter-relationships between paired audiovisual representations. At the inference stage, the preserved audio representations could be extracted from memory by the learned inter-relationships using only video input. Second, a joint cross-fusion model using the attention mechanism could effectively exploit complementary intermodal relationships, and the model calculates cross-attention weights on the basis of the correlations between joint feature representations and individual modalities. Lastly, our proposed model achieved a 4.0% reduction in WER in a -15 dB SNR environment compared to the baseline method, and a 10.1% reduction in WER compared to speech recognition. The experimental results show that our method could achieve a significant improvement over speech recognition models in different noise environments.

Improving Whisper's Recognition Performance for Under-Represented Language Kazakh Leveraging Unpaired Speech and Text

Improving Whispered Speech Recognition Performance using Pseudo-whispered based Data Augmentation

Perceiver-Prompt: Flexible Speaker Adaptation in Whisper for Chinese Disordered Speech Recognition

Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper

A Study of Speech Recognition for Kazakh Based on Unsupervised Pre-Training

Using fine-tuning and min lookahead beam search to improve Whisper

LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR

Whisper Turns Stronger: Augmenting Wav2Vec 2.0 for Superior ASR in Low-Resource Languages

A Multitask Training Approach to Enhance Whisper with Contextual Biasing and Open-Vocabulary Keyword Spotting

M2R-Whisper: Multi-stage and Multi-scale Retrieval Augmentation for Enhancing Whisper

SQ-Whisper: Speaker-Querying based Whisper Model for Target-Speaker ASR

A Multitask Training Approach to Enhance Whisper with Open-Vocabulary Keyword Spotting

Enhancing Whisper Model for Pronunciation Assessment with Multi-Adapters

Enhancing Multilingual ASR for Unseen Languages via Language Embedding Modeling

Efficient Compression of Multitask Multilingual Speech Models

Fine-tuning Whisper on Low-Resource Languages for Real-World Applications

Improving Speech Recognition Performance in Noisy Environments by Enhancing Lip Reading Accuracy

Exploration of Whisper fine-tuning strategies for low-resource ASR

Exploring Native and Non-Native English Child Speech Recognition With Whisper

Weighted Cross-entropy for Low-Resource Languages in Multilingual Speech Recognition

Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling