Abstract:Although great progress has been made in automatic speech recognition (ASR), significant performance degradation still exists in noisy environments. In this paper, a novel factor-aware training framework, named neural network-based multifactor aware joint training, is proposed to improve the recognition accuracy for noise robust speech recognition. This approach is a structured model which integrates several different functional modules into one computational deep model. We explore and extract speaker, phone, and environment factor representations using deep neural networks (DNNs), which are integrated into the main ASR DNN to improve classification accuracy. In addition, the hidden activations in the main ASR DNN are used to improve factor extraction, which in turn helps theASRDNN. All the model parameters, including those in the ASR DNN and factor extraction DNNs, are jointly optimized under the multitask learning framework. Unlike prior traditional techniques for the factor-aware training, our approach requires no explicit separate stages for factor extraction and adaptation. Moreover, the proposed neural network-based multifactor aware joint training can be easily combined with the conventional factor-aware training which uses the explicit factors, such as i-vector, noise energy, and T 60 value to obtain additional improvement. The proposed method is evaluated on two main noise robust tasks: the AMI single distant microphone task in which reverberation is the main concern, and the Aurora4 task in which multiple noise types exist. Experiments on both tasks show that the proposed model can significantly reduce word error rate (WER). The best configuration achieved more than 15% relative reduction in WER over the baselines on these two tasks.

A Binaural Deep Neural Networks Parameter Mask for the Robust Automatic Speech Recognition System

A DNN Parameter Mask for the Binaural Reverberant Speech Segregation

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

A Dual Microphone Speech Enhancement Method With A Smoothing Parameter Mask

Parameter Masks for Close Talk Speech Segregation Using Deep Neural Networks

Binaural Deep Neural Network for Robust Speech Enhancement

Localization Based Stereo Speech Source Separation Using Probabilistic Time-Frequency Masking and Deep Neural Networks

Robust Mask Estimation by Integrating Neural Network-Based and Clustering-Based Approaches for Adaptive Acoustic Beamforming.

Modelling human speech recognition in challenging noise maskers using machine learning

Auditory Features For The Close Talk Speech Enhancement With Parameter Masks

Joint Noise and Mask Aware Training for DNN-based Speech Enhancement with SUB-band Features

Neural Network Based Multi-Factor Aware Joint Training for Robust Speech Recognition.

Single-channel speech enhancement using improved progressive deep neural network and masking-based harmonic regeneration

Using an Adjustment Training and a Smoothing Mask for Speech Segregation

Joint Acoustic Factor Learning for Robust Deep Neural Network Based Automatic Speech Recognition

Improving Robustness of Deep Neural Network Acoustic Models via Speech Separation and Joint Adaptive Training

Robust Beamforming for Speech Recognition Using DNN-Based Time-Frequency Masks Estimation.

SS-BRPE: Self-Supervised Blind Room Parameter Estimation Using Attention Mechanisms

Deep Interaction between Masking and Mapping Targets for Single-Channel Speech Enhancement

Impact of Mask Type as Training Target for Speech Intelligibility and Quality in Cochlear-Implant Noise Reduction

BERP: A Blind Estimator of Room Acoustic and Physical Parameters for Single-Channel Noisy Speech Signals