Abstract:Pruning neural networks has become popular in the last decade when it was shown that a large number of weights can be safely removed from modern neural networks without compromising accuracy. Numerous pruning methods have been proposed since, each claiming to be better than prior art, however, at the cost of increasingly complex pruning methodologies. These methodologies include utilizing importance scores, getting feedback through back-propagation or having heuristics-based pruning rules amongst others. In this work, we question whether this pattern of introducing complexity is really necessary to achieve better pruning results. We benchmark these SOTA techniques against a simple pruning baseline, namely, Global Magnitude Pruning (Global MP), that ranks weights in order of their magnitudes and prunes the smallest ones. Surprisingly, we find that vanilla Global MP performs very well against the SOTA techniques. When considering sparsity-accuracy trade-off, Global MP performs better than all SOTA techniques at all sparsity ratios. When considering FLOPs-accuracy trade-off, some SOTA techniques outperform Global MP at lower sparsity ratios, however, Global MP starts performing well at high sparsity ratios and performs very well at extremely high sparsity ratios. Moreover, we find that a common issue that many pruning algorithms run into at high sparsity rates, namely, layer-collapse, can be easily fixed in Global MP. We explore why layer collapse occurs in networks and how it can be mitigated in Global MP by utilizing a technique called Minimum Threshold. We showcase the above findings on various models (WRN-28-8, ResNet-32, ResNet-50, MobileNet-V1 and FastGRNN) and multiple datasets (CIFAR-10, ImageNet and HAR-2). Code is available at <a class="link-external link-https" href="https://github.com/manasgupta-1/GlobalMP" rel="external noopener nofollow">this https URL</a>.

Pruning inferior systems using subjective constraints with sequentially added thresholds

Selection of the Best in the Presence of Subjective Stochastic Constraints

A Relative Feasibility Degree Based Approach for Constrained Optimization Problems.

Balancing Act: Constraining Disparate Impact in Sparse Models

Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot

An Enhanced Numerical Discretization Method for Transient Stability Constrained Optimal Power Flow

Improving the filtering of Branch-And-Bound MDD solver (extended)

Towards Sobolev Pruning

Multi-Dimensional Pruning: Joint Channel, Layer and Block Pruning with Latency Constraint

Is Complexity Required for Neural Network Pruning? A Case Study on Global Magnitude Pruning

An Analysis of Reduced Error Pruning

Exploring the Performance of Pruning Methods in Neural Networks: An Empirical Study of the Lottery Ticket Hypothesis

A Probabilistic Approach to Neural Network Pruning

Investigating Channel Pruning through Structural Redundancy Reduction -- A Statistical Study

Rethinking the Value of Network Pruning

TT-MPD: Test Time Model Pruning and Distillation

Ensemble Pruning: A Submodular Function Maximization Perspective.

A New Pruning Method for Solving Decision Trees and Game Trees

Coupling Fairness and Pruning in a Single Run: a Bi-level Optimization Perspective

An Efficient Constraint-Based Pruning Method to Improve Chlorine Dosage Optimization