Abstract:A field that has directly benefited from the recent advances in deep learning is Automatic Speech Recognition (ASR). Despite the great achievements of the past decades, however, a natural and robust human-machine speech interaction still appears to be out of reach, especially in challenging environments characterized by significant noise and reverberation. To improve robustness, modern speech recognizers often employ acoustic models based on Recurrent Neural Networks (RNNs), that are naturally able to exploit large time contexts and long-term speech modulations. It is thus of great interest to continue the study of proper techniques for improving the effectiveness of RNNs in processing speech signals. In this paper, we revise one of the most popular RNN models, namely Gated Recurrent Units (GRUs), and propose a simplified architecture that turned out to be very effective for ASR. The contribution of this work is two-fold: First, we analyze the role played by the reset gate, showing that a significant redundancy with the update gate occurs. As a result, we propose to remove the former from the GRU design, leading to a more efficient and compact single-gate model. Second, we propose to replace hyperbolic tangent with ReLU activations. This variation couples well with batch normalization and could help the model learn long-term dependencies without numerical issues. Results show that the proposed architecture, called Light GRU (Li-GRU), not only reduces the per-epoch training time by more than 30% over a standard GRU, but also consistently improves the recognition accuracy across different tasks, input features, noisy conditions, as well as across different ASR paradigms, ranging from standard DNN-HMM speech recognizers to end-to-end CTC models.

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Residual Recurrent Neural Networks for Learning Sequential Representations.

Hierarchically Gated Recurrent Neural Network for Sequence Modeling

Sequence Classification Restricted Boltzmann Machines With Gated Units

Refined Gate: A Simple and Effective Gating Mechanism for Recurrent Units

Investigating gated recurrent neural networks for speech synthesis

Memory Visualization for Gated Recurrent Neural Networks in Speech Recognition

Gates Are Not What You Need in RNNs

A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures

Recurrently Controlled Recurrent Networks

Light Gated Recurrent Units for Speech Recognition

GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling

A Critical Review of Recurrent Neural Networks for Sequence Learning

Minimal Gated Unit for Recurrent Neural Networks

Gating creates slow modes and controls phase-space complexity in GRUs and LSTMs

Gated Recurrent Units Based Hybrid Acoustic Models for Robust Speech Recognition

Gated recurrent units viewed through the lens of continuous time dynamical systems

A Predictive Model for Music Based on Learned Interval Representations

Improving speech recognition by revising gated recurrent units

Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions

Deep Gate Recurrent Neural Network