Abstract:Although great progress has been made in automatic speech recognition (ASR), significant performance degradation still exists in noisy environments. In this paper, a novel factor-aware training framework, named neural network-based multifactor aware joint training, is proposed to improve the recognition accuracy for noise robust speech recognition. This approach is a structured model which integrates several different functional modules into one computational deep model. We explore and extract speaker, phone, and environment factor representations using deep neural networks (DNNs), which are integrated into the main ASR DNN to improve classification accuracy. In addition, the hidden activations in the main ASR DNN are used to improve factor extraction, which in turn helps theASRDNN. All the model parameters, including those in the ASR DNN and factor extraction DNNs, are jointly optimized under the multitask learning framework. Unlike prior traditional techniques for the factor-aware training, our approach requires no explicit separate stages for factor extraction and adaptation. Moreover, the proposed neural network-based multifactor aware joint training can be easily combined with the conventional factor-aware training which uses the explicit factors, such as i-vector, noise energy, and T 60 value to obtain additional improvement. The proposed method is evaluated on two main noise robust tasks: the AMI single distant microphone task in which reverberation is the main concern, and the Aurora4 task in which multiple noise types exist. Experiments on both tasks show that the proposed model can significantly reduce word error rate (WER). The best configuration achieved more than 15% relative reduction in WER over the baselines on these two tasks.

Dbn Based Multi-Stream Models For Speech

Multi-stream Asynchrony Modeling for Audio-Visual Speech Recognition

Audio Visual Speech Recognition Based on Multi-Stream DBN Models with Articulatory Features

Single Stream Dbn Model For Continuous Speech Recognition And Phone Segmentation

DBN based models for audio-visual speech analysis and recognition

Audio-Visual Tibetan Speech Recognition Based On A Deep Dynamic Bayesian Network For Natural Human Robot Interaction Regular Paper

A Dynamic Bayesian Network Based Framework for Continuous Speech Recognition and Its Token Passing Model

Audio Visual Emotion Recognition Based On Triple-Stream Dynamic Bayesian Network Models

Multi-task Joint-Learning of Deep Neural Networks for Robust Speech Recognition

Novel Articulatory Feature based Dynamic Bayesian Network model for speech recognition

Single Channel Speech Enhancement Algorithm Based on BLSTM-DNN Bidirectional Optimized Hybrid Model

Automatic Speaker Recognition Using Dynamic Bayesian Network.

Tibetan Language Continuous Speech Recognition Based on Dynamic Bayesian Network

Dynamic bayesian networks for audio-visual speaker recognition

Speech-enhanced and Noise-aware Networks for Robust Speech Recognition

Single Stream DBN Model Based Triphone for Continuous Speech Recognition

Densely Connected Multi-Stage Model with Channel Wise Subband Feature for Real-Time Speech Enhancement.

Neural Network Based Multi-Factor Aware Joint Training for Robust Speech Recognition.

International Journal of Advanced Robotic Systems Audio-Visual Tibetan Speech Recognition Based on a Deep Dynamic Bayesian Network for Natural Human Robot Interaction Regular Paper

Acoustic Modeling with Densely Connected Residual Network for Multichannel Speech Recognition