Abstract:Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR), which leverages the rich linguistic knowledge and powerful reasoning ability of LLMs to improve recognition results. The latest work proposes a GER benchmark with HyPoradise dataset to learn the mapping from ASR N-best hypotheses to ground-truth transcription by efficient LLM finetuning, which shows great effectiveness but lacks specificity on noise-robust ASR. In this work, we extend the benchmark to noisy conditions and investigate if we can teach LLMs to perform denoising for GER just like what robust ASR do}, where one solution is introducing noise information as a conditioner into LLM. However, directly incorporating noise embeddings from audio encoder could harm the LLM tuning due to cross-modality gap. To this end, we propose to extract a language-space noise embedding from the N-best list to represent the noise conditions of source speech, which can promote the denoising process in GER. Furthermore, in order to enhance its representation ability of audio noise, we design a knowledge distillation (KD) approach via mutual information estimation to distill the real noise information in audio embeddings to our language embedding. Experiments on various latest LLMs demonstrate our approach achieves a new breakthrough with up to 53.9% correction improvement in terms of word error rate while with limited training data. Analysis shows that our language-space noise embedding can well represent the noise conditions of source speech, under which off-the-shelf LLMs show strong ability of language-space denoising.

Learning Noise Adapters for Incremental Speech Enhancement

Noise Adaptive Speech Enhancement Using Domain Adversarial Training.

Domain Adaptation Using Suitable Pseudo Labels for Speech Enhancement and Dereverberation

Audio Enhancement for Computer Audition -- An Iterative Training Paradigm Using Sample Importance

An New Approach for Incremental Speaker Adaptation

Audio Enhancement for Computer Audition—An Iterative Training Paradigm Using Sample Importance

A Progressive Learning Approach to Adaptive Noise and Speech Estimation for Speech Enhancement and Noisy Speech Recognition.

INCREMENTAL LEARNING FOR END-TO-END AUTOMATIC SPEECH RECOGNITION

Dynamic noise aware training for speech enhancement based on deep neural networks.

Dynamic Noise Embedding: Noise Aware Training and Adaptation for Speech Enhancement

Incremental Learning Algorithm for Sound Event Detection

Remixed2Remixed: Domain adaptation for speech enhancement by Noise2Noise learning with Remixing

Noise-aware Speech Enhancement using Diffusion Probabilistic Model

DIDA: Denoised Imitation Learning based on Domain Adaptation

Large Language Models are Efficient Learners of Noise-Robust Speech Recognition

Effective Noise-aware Data Simulation for Domain-adaptive Speech Enhancement Leveraging Dynamic Stochastic Perturbation

Adapting noisy speech models — Extended uncertainty decoding

Improving Deep Neural Network Based Speech Enhancement in Low SNR Environments

Unsupervised Noise adaptation using Data Simulation

Configurable DOA Estimation using Incremental Learning

Deep Noise Tracking Network: A Hybrid Signal Processing/Deep Learning Approach to Speech Enhancement