Abstract:Although great progress has been made in automatic speech recognition (ASR), significant performance degradation still exists in noisy environments. In this paper, a novel factor-aware training framework, named neural network-based multifactor aware joint training, is proposed to improve the recognition accuracy for noise robust speech recognition. This approach is a structured model which integrates several different functional modules into one computational deep model. We explore and extract speaker, phone, and environment factor representations using deep neural networks (DNNs), which are integrated into the main ASR DNN to improve classification accuracy. In addition, the hidden activations in the main ASR DNN are used to improve factor extraction, which in turn helps theASRDNN. All the model parameters, including those in the ASR DNN and factor extraction DNNs, are jointly optimized under the multitask learning framework. Unlike prior traditional techniques for the factor-aware training, our approach requires no explicit separate stages for factor extraction and adaptation. Moreover, the proposed neural network-based multifactor aware joint training can be easily combined with the conventional factor-aware training which uses the explicit factors, such as i-vector, noise energy, and T 60 value to obtain additional improvement. The proposed method is evaluated on two main noise robust tasks: the AMI single distant microphone task in which reverberation is the main concern, and the Aurora4 task in which multiple noise types exist. Experiments on both tasks show that the proposed model can significantly reduce word error rate (WER). The best configuration achieved more than 15% relative reduction in WER over the baselines on these two tasks.

Joint Training Of Complex Ratio Mask Based Beamformer And Acoustic Model For Noise Robust Asr

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

Robust Mask Estimation by Integrating Neural Network-Based and Clustering-Based Approaches for Adaptive Acoustic Beamforming.

Beamforming-based Speech Enhancement Based on Optimal Ratio Mask

Neural Mask based Multi-channel Convolutional Beamforming for Joint Dereverberation, Echo Cancellation and Denoising

Deep Long Short-Term Memory Adaptive Beamforming Networks For Multichannel Robust Speech Recognition

Multi-channel Speech Enhancement Based on the MVDR Beamformer and Postfilter.

Neural Network Based Multi-Factor Aware Joint Training for Robust Speech Recognition.

An Iterative Mask Estimation Approach to Deep Learning Based Multi-Channel Speech Recognition

Neural Network Adaptive Beamforming for Robust Multichannel Speech Recognition

4D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoders

4D ASR: Joint Beam Search Integrating CTC, Attention, Transducer, and Mask Predict Decoders

Neural Spatio-Temporal Beamformer for Target Speech Separation

Joint Beamforming and Speaker-Attributed ASR for Real Distant-Microphone Meeting Transcription

An efficient joint training model for monaural noisy-reverberant speech recognition

Masking-based Neural Beamformer for Multichannel Speech Enhancement

Coarse-Grained Attention Fusion with Joint Training Framework for Complex Speech Enhancement and End-to-End Speech Recognition

Unsupervised Speech Enhancement Based on Multichannel NMF-Informed Beamforming for Noise-Robust Automatic Speech Recognition

Improving Robustness of Deep Neural Network Acoustic Models via Speech Separation and Joint Adaptive Training

Attention-Based Beamformer For Multi-Channel Speech Enhancement

A Speech Enhancement Algorithm By Iterating Single- And Multi-Microphone Processing And Its Application To Robust Asr