Abstract:To model time-varying nonlinear temporal dynamics in sequential data, a recurrent network capable of varying and adjusting the recurrence depth between input intervals is examined. The recurrence depth is extended by several intermediate hidden state units, and the weight parameters involved in determining these units are dynamically calculated. The motivation behind the paper lies on overcoming a deficiency in Recurrent Highway Networks and improving their performances which are currently at the forefront of RNNs: 1) Determining the appropriate number of recurrent depth in RHN for different tasks is a huge burden and just setting it to a large number is computationally wasteful with possible repercussion in terms of performance degradation and high latency. Expanding on the idea of adaptive computation time (ACT), with the use of an elastic gate in the form of a rectified exponentially decreasing function taking on as arguments as previous hidden state and input, the proposed model is able to evaluate the appropriate recurrent depth for each input. The rectified gating function enables the most significant intermediate hidden state updates to come early such that significant performance gain is achieved early. 2) Updating the weights from that of previous intermediate layer offers a richer representation than the use of shared weights across all intermediate recurrence layers. The weight update procedure is just an expansion of the idea underlying hypernetworks. To substantiate the effectiveness of the proposed network, we conducted three experiments: regression on synthetic data, human activity recognition, and language modeling on the Penn Treebank dataset. The proposed networks showed better performance than other state-of-the-art recurrent networks in all three experiments.

A Recalling-Enhanced Recurrent Neural Network: Conjugate Gradient Learning Algorithm and Its Convergence Analysis

An Efficient Elman Neural Networks Based on Improved Conjugate Gradient Method with Generalized Armijo Search.

A Modified Conjugate Gradient-Based Elman Neural Network

Residual Recurrent Neural Networks for Learning Sequential Representations.

Convergence Analysis of Regularized Elman Neural Networks under Relaxed Conditions

Equilibrated Recurrent Neural Network: Neuronal Time-Delayed Self-Feedback Improves Accuracy and Stability

Convergence of Gradient Method for A Fully Recurrent Neural Network

An Improvement on Recurrent Neural Network by Combining Convolution Neural Network and a Simple Initialization of the Weights

On extended long short-term memory and dependent bidirectional recurrent neural network

An Augmented Lagrangian Method for Training Recurrent Neural Networks

Early Improving Recurrent Elastic Highway Network

Memory Recall: A Simple Neural Network Training Framework Against Catastrophic Forgetting

A Modified Elman Neural Network with a New Learning Rate Scheme.

A Novel Conjugate Gradient Method with Generalized Armijo Search for Efficient Training of Feedforward Neural Networks

An Adaptive Growing and Pruning Algorithm for Designing Recurrent Neural Network.

An Algorithm Of Supervised Learning For Elman Neural Network

Learning Longer Memory in Recurrent Neural Networks

Investigating Recurrent Neural Network Memory Structures using Neuro-Evolution

General Recurrent Neural Network for Solving Generalized Linear Matrix Equation

Convergence of Gradient Method for Elman Networks

Improving Learning Efficiency of Recurrent Neural Network Through Adjusting Weights of All Layers in a Biologically-Inspired Framework.