Abstract:Expansion property of a graph refers to its strong connectivity as well as sparseness. It has been reported that deep neural networks can be pruned to a high degree of sparsity while maintaining their performance. Such pruning is essential for performing real time sequence learning tasks using recurrent neural networks in resource constrained platforms. We prune recurrent networks such as RNNs and LSTMs, maintaining a large spectral gap of the underlying graphs and ensuring their layerwise expansion properties. We also study the time unfolded recurrent network graphs in terms of the properties of their bipartite layers. Experimental results for the benchmark sequence MNIST, CIFAR-10, and Google speech command data show that expander graph properties are key to preserving classification accuracy of RNN and LSTM.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper mainly explores how to maintain performance in pruned Recurrent Neural Networks (RNNs) and Long - Short Term Memory Networks (LSTMs), especially by maintaining the graph - expansion properties of these networks. Specifically, the authors attempt to solve the following key problems: 1. **Real - time sequence learning tasks on resource - constrained platforms**: - RNNs and LSTMs require a large amount of computational resources when processing sequence data, which makes them difficult to deploy on resource - constrained platforms. Therefore, how to reduce the number of parameters and computational complexity of these networks without significantly degrading performance is an important research topic. 2. **Effectiveness of pruning methods**: - Although there are already some pruning methods that can reduce the number of network parameters, these methods are not very effective when applied to recurrent networks. In particular, there are still challenges in achieving high sparsity while maintaining high task accuracy. 3. **Relationship between graph - expansion properties and network performance**: - The authors assume that the expander properties of the graph are crucial for maintaining the performance of the pruned network. They attempt to verify this hypothesis and explore how to use these properties to guide the pruning process. 4. **Importance of different connection types**: - Research the impact of connections between different layers (such as input - to - hidden layer, feedback layer, hidden - to - output layer) on the performance of the pruned network, thereby providing guidance for pruning strategies. 5. **Noise robustness**: - Explore the performance of the pruned network in a noisy environment, ensuring that the pruned model can not only perform well on clean data but also maintain a high classification accuracy on noisy data. ### Main contributions - **Proposed a method for analyzing RNN/LSTM architectures based on graph - expansion properties**. - **Determined the eligibility criteria based on expansion properties to identify lightweight RNN/LSTM architectures suitable for real - time hardware applications**. - **Experimental observations show that even when the network density is significantly reduced, RNN/LSTM architectures that satisfy the properties of the layered bipartite Ramanujan graph can still maintain high accuracy**. - **Verified the effectiveness of expansion - property - based techniques in analyzing and testing the applicability of lightweight architectures**. - **Discovered the importance of different types of connections between different layers, providing guidance for pruning decisions**. - **Proposed expansion - graph properties that can be used as stopping conditions for RNN and LSTM pruning algorithms**. Through these studies, the authors provide theoretical and experimental support for achieving efficient and performance - stable sparse RNNs and LSTMs, which helps to promote the application of deep - learning technology on resource - constrained devices.

Graph Expansion in Pruned Recurrent Neural Network Layers Preserve Performance

Residual Recurrent Neural Networks for Learning Sequential Representations.

Investigating Sparsity in Recurrent Neural Networks

Stage-Wise Magnitude-Based Pruning for Recurrent Neural Networks

Adversarial Structured Neural Network Pruning

Graph Neural Networks Exponentially Lose Expressive Power for Node Classification

Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN

DepGraph: Towards Any Structural Pruning

Structured Pruning of Recurrent Neural Networks through Neuron Selection

Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics

Pruning graph neural networks by evaluating edge properties

Pruning of Deep Spiking Neural Networks Through Gradient Rewiring.

Efficient recurrent architectures through activity sparsity and sparse back-propagation through time

N2NSkip: Learning Highly Sparse Networks using Neuron-to-Neuron Skip Connections

Feature Expansion for Graph Neural Networks

Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity

Rethinking Graph Lottery Tickets: Graph Sparsity Matters

A Generalization of Continuous Relaxation in Structured Pruning

Spectral Graph Pruning Against Over-Squashing and Over-Smoothing

Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning