Abstract:Since sparse neural networks usually contain many zero weights, these unnecessary network connections can potentially be eliminated without degrading network performance. Therefore, well-designed sparse neural networks have the potential to significantly reduce FLOPs and computational resources. In this work, we propose a new automatic pruning method - Sparse Connectivity Learning (SCL). Specifically, a weight is re-parameterized as an element-wise multiplication of a trainable weight variable and a binary mask. Thus, network connectivity is fully described by the binary mask, which is modulated by a unit step function. We theoretically prove the fundamental principle of using a straight-through estimator (STE) for network pruning. This principle is that the proxy gradients of STE should be positive, ensuring that mask variables converge at their minima. After finding Leaky ReLU, Softplus, and Identity STEs can satisfy this principle, we propose to adopt Identity STE in SCL for discrete mask relaxation. We find that mask gradients of different features are very unbalanced, hence, we propose to normalize mask gradients of each feature to optimize mask variable training. In order to automatically train sparse masks, we include the total number of network connections as a regularization term in our objective function. As SCL does not require pruning criteria or hyper-parameters defined by designers for network layers, the network is explored in a larger hypothesis space to achieve optimized sparse connectivity for the best performance. SCL overcomes the limitations of existing automatic pruning methods. Experimental results demonstrate that SCL can automatically learn and select important network connections for various baseline network structures. Deep learning models trained by SCL outperform the SOTA human-designed and automatic pruning methods in sparsity, accuracy, and FLOPs reduction.

Growing Sparse Neural Networks

Efficient Structure Slimming for Spiking Neural Networks

Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science

Fusing functional connectivity with network nodal information for sparse network pattern learning of functional brain networks

Explore the Knowledge contained in Network Weights to Obtain Sparse Neural Networks

Automatic Sparse Connectivity Learning for Neural Networks

N2NSkip: Learning Highly Sparse Networks using Neuron-to-Neuron Skip Connections

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

Training Sparse Neural Network by Constraining Synaptic Weight on Unit Lp Sphere

ESL-SNNs: An Evolutionary Structure Learning Strategy for Spiking Neural Networks

Pursing the Sparse Limitation of Spiking Deep Learning Structures

State Transition of Dendritic Spines Improves Learning of Sparse Spiking Neural Networks.

Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity

Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networks

Sparse optimization guided pruning for neural networks

Always-Sparse Training by Growing Connections with Guided Stochastic Exploration

Pruning of Deep Spiking Neural Networks Through Gradient Rewiring.

Students and teachers learning together: a robust training strategy for neural network pruning

Jointly Learning Network Connections and Link Weights in Spiking Neural Networks

Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN

Multiple Sparse Spaces Network Pruning Via a Joint Similarity Criterion