Abstract:Although great progress has been made in automatic speech recognition, significant performance degradation still exists in noisy environments. Our previous work has demonstrated the superior noise robustness of very deep convolutional neural networks (VDCNN). Based on our work on VDCNNs, this paper proposes a more advanced model referred to as the very deep convolutional residual network (VDCRN). This new model incorporates batch normalization and residual learning, showing more robustness than previous VDCNNs. Then, to alleviate the mismatch between the training and testing conditions, model adaptation and adaptive training are developed and compared for the new VDCRN. This paper focuses on factor aware training (FAT) and cluster adaptive training (CAT). For FAT, a unified framework is explored. For CAT, two schemes are first explored to construct the bases in the canonical model; furthermore, a factorized version of CAT is designed to address multiple nonspeech variabilities in one model. Finally, a complete multipass system is proposed to achieve the best system performance in the noisy scenarios. The proposed new approaches are evaluated on three different tasks: Aurora4 (simulated data with additive noise and channel distortion), CHiME4 (both simulated and real data with additive noise and reverberation), and the AMI meeting transcription task (real data with significant reverberation). The evaluation not only includes different noisy conditions, but also covers both simulated and real noisy data. The experiments show that the new VDCRN is more robust, and the adaptation on this model can further significantly reduce the word error rate (WER). The proposed best architecture obtains consistent and very large improvements on all tasks compared to the baseline VDCNN or long short-term memory. Particularly, on Aurora4 a new milestone 5.67% WER is achieved by only improving acoustic modeling.

Empirically Combining Unnormalized NNLM and Back-off N -Gram for Fast N -Best Rescoring in Speech Recognition

Efficient One-Pass Decoding with Nnlm for Speech Recognition

Lattice Rescoring Based on Large Ensemble of Complementary Neural Language Models

Variance regularization of RNNLM for speech recognition

Acceleration Strategies for Speech Recognition Based on Deep Neural Networks

Class-Based Neural Network Language Model for Second-Pass Rescoring in ASR

Improving Accented Mandarin Speech Recognition by Using Recurrent Neural Network Based Language Model Adaptation

Context-aware RNNLM Rescoring for Conversational Speech Recognition

Incoherent training of deep neural networks to de-correlate bottleneck features for speech recognition

Improving deep neural networks for LVCSR using dropout and shrinking structure

Neural Lattice Search for Speech Recognition.

Recurrent Neural Network Based Language Model Adaptation for Accent Mandarin Speech.

Advanced Recurrent Network-Based Hybrid Acoustic Models for Low Resource Speech Recognition

A Cluster-Based Multiple Deep Neural Networks Method for Large Vocabulary Continuous Speech Recognition

State-Clustering Based Multiple Deep Neural Networks Modeling Approach for Speech Recognition

Combining Hybrid DNN-HMM ASR Systems with Attention-Based Models Using Lattice Rescoring

Improving the Decoding Efficiency of Deep Neural Network Acoustic Models by Cluster-Based Senone Selection.

Neural Network Based Multi-Factor Aware Joint Training for Robust Speech Recognition.

Adaptive Very Deep Convolutional Residual Network for Noise Robust Speech Recognition

Single-Channel Speech Enhancement Algorithm Based on ME-MGCRN in Low Signal-to-Noise Scenario

Deep Recurrent Convolutional Neural Network: Improving Performance For Speech Recognition