Abstract:We propose an integrated end-to-end automatic speech recognition (ASR) paradigm by joint learning of the front-end speech signal processing and back-end acoustic modeling. We believe that only good signal processing can lead to top ASR performance in challenging acoustic environments. This notion leads to a unified deep neural network (DNN) framework for distant speech processing that can achieve both high-quality enhanced speech and high-accuracy ASR simultaneously. Our goal is accomplished by two techniques, namely: (i) a reverberation-time-aware DNN based speech dereverberation architecture that can handle a wide range of reverberation times to enhance speech quality of reverberant and noisy speech, followed by (ii) DNN-based multicondition training that takes both clean-condition and multicondition speech into consideration, leveraging upon an exploitation of the data acquired and processed with multichannel microphone arrays, to improve ASR performance. The final end-to-end system is established by a joint optimization of the speech enhancement and recognition DNNs. The recent REverberant Voice Enhancement and Recognition Benchmark (REVERB) Challenge task is used as a test bed for evaluating our proposed framework. We first report on superior objective measures in enhanced speech to those listed in the 2014 REVERB Challenge Workshop on the simulated data test set. Moreover, we obtain the best single-system word error rate (WER) of 13.28 on the 1-channel REVERB simulated data with the proposed DNN-based pre-processing algorithm and clean-condition training. Leveraging upon joint training with more discriminative ASR features and improved neural network based language models, a low single-system WER of 4.46 is attained. Next, a new multi-channel-condition joint learning and testing scheme delivers a state-of-the-art WER of 3.76 on the 8-channel simulated data with a single ASR system. Finally, we also report on a preliminary yet promising experimentation with the REVERB real test data.

Auxiliary Features from Laser-Doppler Vibrometer Sensor for Deep Neural Network Based Robust Speech Recognition

An End-to-End Deep Learning Approach to Simultaneous Speech Dereverberation and Acoustic Modeling for Robust Speech Recognition

Auxiliary Multimodal LSTM for Audio-visual Speech Recognition and Lipreading

Improving Robustness of Deep Neural Network Acoustic Models via Speech Separation and Joint Adaptive Training

An Experimental Study on Speech Enhancement Based on Deep Neural Networks

Robust Speech Recognition With Speech Enhanced Deep Neural Networks

DLD: An Optimized Chinese Speech Recognition Model Based on Deep Learning

Building DNN acoustic models for large vocabulary speech recognition

Improving Deep Neural Network Based Speech Enhancement in Low SNR Environments

Modeling Speaker Variability Using Long Short-Term Memory Networks For Speech Recognition

Toward a Better Understanding of Deep Neural Network Based Acoustic Modelling: An Empirical Investigation

A Regression Approach to Single-Channel Speech Separation Via High-Resolution Deep Neural Networks.

A regression approach to speech enhancement based on deep neural networks

A Universal VAD Based on Jointly Trained Deep Neural Networks.

Deep Belief Networks Based Voice Activity Detection

Joint Training of DNNs by Incorporating an Explicit Dereverberation Structure for Distant Speech Recognition

Noise Robust Speech Recognition on Aurora4 by Humans and Machines.

Multiple-target Deep Learning for LSTM-RNN Based Speech Enhancement

Improving Deep Neural Networks Based Speaker Verification Using Unlabeled Data

Investigation of deep neural networks (DNN) for large vocabulary continuous speech recognition: Why DNN surpasses GMMS in acoustic modeling

Joint Training Of Front-End And Back-End Deep Neural Networks For Robust Speech Recognition