Abstract:Neural network pruning is a fruitful area of research with surging interest in high sparsity regimes. Benchmarking in this domain heavily relies on faithful representation of the sparsity of subnetworks, which has been traditionally computed as the fraction of removed connections (direct sparsity). This definition, however, fails to recognize unpruned parameters that detached from input or output layers of underlying subnetworks, potentially underestimating actual effective sparsity: the fraction of inactivated connections. While this effect might be negligible for moderately pruned networks (up to 10-100 compression rates), we find that it plays an increasing role for thinner subnetworks, greatly distorting comparison between different pruning algorithms. For example, we show that effective compression of a randomly pruned LeNet-300-100 can be orders of magnitude larger than its direct counterpart, while no discrepancy is ever observed when using SynFlow for pruning [Tanaka et al., 2020]. In this work, we adopt the lens of effective sparsity to reevaluate several recent pruning algorithms on common benchmark architectures (e.g., LeNet-300-100, VGG-19, ResNet-18) and discover that their absolute and relative performance changes dramatically in this new and more appropriate framework. To aim for effective, rather than direct, sparsity, we develop a low-cost extension to most pruning algorithms. Further, equipped with effective sparsity as a reference frame, we partially reconfirm that random pruning with appropriate sparsity allocation across layers performs as well or better than more sophisticated algorithms for pruning at initialization [Su et al., 2020]. In response to this observation, using a simple analogy of pressure distribution in coupled cylinders from physics, we design novel layerwise sparsity quotas that outperform all existing baselines in the context of random pruning.

CRESPR: Modular sparsification of DNNs to improve pruning performance and model interpretability

Class-Aware Pruning for Efficient Neural Networks

Structured Pruning for Efficient Convolutional Neural Networks Via Incremental Regularization

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

A Pruning Method Based on the Dissimilarity of Angle among Channels and Filters

A roulette wheel-based pruning method to simplify cumbersome deep neural networks

Multiple Sparse Spaces Network Pruning Via a Joint Similarity Criterion

Structural Pruning in Deep Neural Networks: A Small-World Approach

Global Sparse Momentum SGD for Pruning Very Deep Neural Networks

DARB: A Density-Aware Regular-Block Pruning for Deep Neural Networks

Dual-Grained Lightweight Strategy

CRISP: Hybrid Structured Sparsity for Class-aware Model Pruning

A Dynamic Pruning Method on Multiple Sparse Structures in Deep Neural Networks

State Transition of Dendritic Spines Improves Learning of Sparse Spiking Neural Networks.

SR-init: an Interpretable Layer Pruning Method

One-Cycle Pruning: Pruning ConvNets Under a Tight Training Budget

Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

Three-Stage Global Channel Pruning for Resources-Limited Platform

Efficient DNN Neuron Pruning by Minimizing Layer-wise Nonlinear Reconstruction Error

(Pen-) Ultimate DNN Pruning

Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity