Abstract:A field that has directly benefited from the recent advances in deep learning is Automatic Speech Recognition (ASR). Despite the great achievements of the past decades, however, a natural and robust human-machine speech interaction still appears to be out of reach, especially in challenging environments characterized by significant noise and reverberation. To improve robustness, modern speech recognizers often employ acoustic models based on Recurrent Neural Networks (RNNs), that are naturally able to exploit large time contexts and long-term speech modulations. It is thus of great interest to continue the study of proper techniques for improving the effectiveness of RNNs in processing speech signals. In this paper, we revise one of the most popular RNN models, namely Gated Recurrent Units (GRUs), and propose a simplified architecture that turned out to be very effective for ASR. The contribution of this work is two-fold: First, we analyze the role played by the reset gate, showing that a significant redundancy with the update gate occurs. As a result, we propose to remove the former from the GRU design, leading to a more efficient and compact single-gate model. Second, we propose to replace hyperbolic tangent with ReLU activations. This variation couples well with batch normalization and could help the model learn long-term dependencies without numerical issues. Results show that the proposed architecture, called Light GRU (Li-GRU), not only reduces the per-epoch training time by more than 30% over a standard GRU, but also consistently improves the recognition accuracy across different tasks, input features, noisy conditions, as well as across different ASR paradigms, ranging from standard DNN-HMM speech recognizers to end-to-end CTC models.

Gate Activation Signal Analysis for Gated Recurrent Neural Networks and Its Correlation with Phoneme Boundaries.

Analysis on Gated Recurrent Unit based Question Detection Approach

Investigating gated recurrent neural networks for speech synthesis

Hyper-Gated Recurrent Neural Networks for Chinese Word Segmentation.

Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions

Refined Gate: A Simple and Effective Gating Mechanism for Recurrent Units

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Memory Visualization for Gated Recurrent Neural Networks in Speech Recognition

Time-domain Analysis of Neural Tracking of Hierarchical Linguistic Structures

Occam's Gates

Dependency-based Gated Recursive Neural Network for Chinese Word Segmentation.

Question Detection from Acoustic Features Using Recurrent Neural Network with Gated Recurrent Unit

Light Gated Recurrent Units for Speech Recognition

Segment Boundary Detection via Class Entropy Measurements in Connectionist Phoneme Recognition

Chinese dialect tone’s recognition using gated spiking neural P systems

A Spiking Neural Network Model for Sound Recognition.

Representation of linguistic form and function in recurrent neural networks

Gated parametric neuron for spike-based audio recognition

Auditory Sensory Gating: Effects of Noise

Gated Word-Character Recurrent Language Model

Deciphering Phonemes from Syllables in Blood Oxygenation Level-Dependent Signals in Human Superior Temporal Gyrus