Abstract:In the past few years, neural networks have evolved from simple Feedforward Neural Networks to more complex neural networks, such as Convolutional Neural Networks and Recurrent Neural Networks. Where CNNs are a perfect fit for tasks where the sequence is not important such as image recognition, RNNs are useful when order is important such as machine translation. An increasing number of layers in a neural network is one way to improve its performance, but it also increases its complexity making it much more time and power-consuming to train. One way to tackle this problem is to introduce sparsity in the architecture of the neural network. Pruning is one of the many methods to make a neural network architecture sparse by clipping out weights below a certain threshold while keeping the performance near to the original. Another way is to generate arbitrary structures using random graphs and embed them between an input and output layer of an Artificial Neural Network. Many researchers in past years have focused on pruning mainly CNNs, while hardly any research is done for the same in RNNs. The same also holds in creating sparse architectures for RNNs by generating and embedding arbitrary structures. Therefore, this thesis focuses on investigating the effects of the before-mentioned two techniques on the performance of RNNs. We first describe the pruning of RNNs, its impact on the performance of RNNs, and the number of training epochs required to regain accuracy after the pruning is performed. Next, we continue with the creation and training of Sparse Recurrent Neural Networks and identify the relation between the performance and the graph properties of its underlying arbitrary structure. We perform these experiments on RNN with Tanh nonlinearity (RNN-Tanh), RNN with ReLU nonlinearity (RNN-ReLU), GRU, and LSTM. Finally, we analyze and discuss the results achieved from both the experiments.

Rethinking the Relationship between Recurrent and Non-Recurrent Neural Networks: A Study in Sparsity

Residual Recurrent Neural Networks for Learning Sequential Representations.

Towards Interpreting Recurrent Neural Networks Through Probabilistic Abstraction

Neural Networks: Computational Models and Applications - ReadingSample

Investigating Sparsity in Recurrent Neural Networks

Reversible Recurrent Neural Networks

Universal structural patterns in sparse recurrent neural networks

Recurrent Neural Networks and Long Short-Term Memory Networks: Tutorial and Survey

Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics

Recurrent Neural Networks (RNNs): A gentle Introduction and Overview

Inferring stochastic low-rank recurrent neural networks from neural data

Identification of Recurrent Dynamics in Distributed Neural Populations

Where is all the nonlinearity: flexible nonlinear modeling of behaviorally relevant neural dynamics using recurrent neural networks

Recurrent Network Models Of Sequence Generation And Memory

Path Space for Recurrent Neural Networks with ReLU Activations

Towards a Unified Recurrent Neural Network Theory: the Uniformly Pseudo-Projection-anti-monotone Net

Modeling Neural Representation of Social Recognition Information with Recurrent Neural Networks

The Uncanny Similarity of Recurrence and Depth

Block-Sparse Recurrent Neural Networks

Learning The Sequential Temporal Information with Recurrent Neural Networks

Probing the Relationship Between Latent Linear Dynamical Systems and Low-Rank Recurrent Neural Network Models