Abstract:Although great progress has been made in automatic speech recognition (ASR), significant performance degradation still exists in noisy environments. In this paper, a novel factor-aware training framework, named neural network-based multifactor aware joint training, is proposed to improve the recognition accuracy for noise robust speech recognition. This approach is a structured model which integrates several different functional modules into one computational deep model. We explore and extract speaker, phone, and environment factor representations using deep neural networks (DNNs), which are integrated into the main ASR DNN to improve classification accuracy. In addition, the hidden activations in the main ASR DNN are used to improve factor extraction, which in turn helps theASRDNN. All the model parameters, including those in the ASR DNN and factor extraction DNNs, are jointly optimized under the multitask learning framework. Unlike prior traditional techniques for the factor-aware training, our approach requires no explicit separate stages for factor extraction and adaptation. Moreover, the proposed neural network-based multifactor aware joint training can be easily combined with the conventional factor-aware training which uses the explicit factors, such as i-vector, noise energy, and T 60 value to obtain additional improvement. The proposed method is evaluated on two main noise robust tasks: the AMI single distant microphone task in which reverberation is the main concern, and the Aurora4 task in which multiple noise types exist. Experiments on both tasks show that the proposed model can significantly reduce word error rate (WER). The best configuration achieved more than 15% relative reduction in WER over the baselines on these two tasks.

Feature Denoising Using Joint Sparse Representation for In-Car Speech Recognition.

DENOISPEECH: DENOISING TEXT TO SPEECH WITH FRAME-LEVEL NOISE MODELING

Joint sparse representation based cepstral-domain dereverberation for distant-talking speech recognition

A Solution to Residual Noise in Speech Denoising with Sparse Representation

Supervised Monaural Speech Enhancement Using Complementary Joint Sparse Representations.

Supervised Monaural Speech Enhancement Using Two-Level Complementary Joint Sparse Representations

Joint Training for Simultaneous Speech Denoising and Dereverberation with Deep Embedding Representations

Spectrum Enhancement with Sparse Coding for Robust Speech Recognition

Multi-task Joint-Learning of Deep Neural Networks for Robust Speech Recognition

Supervised Single-Channel Speech Dereverberation and Denoising Using a Two-Stage Model Based Sparse Representation.

Non-linear Spectral Contrast Stretching for In-Car Speech Recognition.

Multitask Sparse Nonnegative Matrix Factorization for Joint Spectral–Spatial Hyperspectral Imagery Denoising

SpatialNet: Extensively Learning Spatial Information for Multichannel Joint Speech Separation, Denoising and Dereverberation

speech and noise dual-stream spectrogram refine network with speech distortion loss for robust speech recognition

Neural Network Based Multi-Factor Aware Joint Training for Robust Speech Recognition.

Low-SNR Speech Enhancement and Separation in Driving Environment

Simultaneous Denoising and Dereverberation Using Deep Embedding Features

A feature-level full-reference image denoising quality assessment method based on joint sparse representation

Sub-band Based Log-Energy and Its Dynamic Range Stretching for Robust In-Car Speech Recognition

Robust In-Car Speech Recognition Based on Nonlinear Multiple Regressions

DualSep: A Light-weight dual-encoder convolutional recurrent network for real-time in-car speech separation