Abstract:Pruning neural networks has become popular in the last decade when it was shown that a large number of weights can be safely removed from modern neural networks without compromising accuracy. Numerous pruning methods have been proposed since then, each claiming to be better than the previous. Many state-of-the-art (SOTA) techniques today rely on complex pruning methodologies utilizing importance scores, getting feedback through back-propagation or having heuristics-based pruning rules amongst others. We question this pattern of introducing complexity in order to achieve better pruning results. We benchmark these SOTA techniques against Global Magnitude Pruning (Global MP), a naive pruning baseline, to evaluate whether complexity is really needed to achieve higher performance. Global MP ranks weights in order of their magnitudes and prunes the smallest ones. Hence, in its vanilla form, it is one of the simplest pruning techniques. Surprisingly, we find that vanilla Global MP outperforms all the other SOTA techniques and achieves a new SOTA result. It also achieves good performance on FLOPs sparsification, which we find is enhanced, when pruning is conducted in a gradual fashion. We also find that Global MP is generalizable across tasks, datasets and models with superior performance. Moreover, a common issue that many pruning algorithms run into at high sparsity rates, namely, layer-collapse, can be easily fixed in Global MP by setting a minimum threshold of weights to be retained in each layer. Lastly, unlike many other SOTA techniques, Global MP does not require any additional algorithm specific hyper-parameters and is very straightforward to tune and implement. We showcase our findings on various models (WRN-28-8, ResNet-32, ResNet-50, MobileNet-V1 and FastGRNN) and multiple datasets (CIFAR-10, ImageNet and HAR-2). Code is available at https://github.com/manasgupta-1/GlobalMP.

Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning

Class-Aware Pruning for Efficient Neural Networks

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

Layer-adaptive sparsity for the Magnitude-based Pruning

An Operator Theoretic View on Pruning Deep Neural Networks

Optimization based Layer-wise Magnitude-based Pruning for DNN Compression

LSOP: Layer-Scaled One-shot Pruning

A Pruning Method Based on the Dissimilarity of Angle among Channels and Filters

Is Complexity Required for Neural Network Pruning? A Case Study on Global Magnitude Pruning

Confident magnitude-based neural network pruning

Generalization Bounds for Magnitude-Based Pruning via Sparse Matrix Sketching

Multi-Dimensional Pruning: Joint Channel, Layer and Block Pruning with Latency Constraint

Sparse optimization guided pruning for neural networks

Structural Pruning via Latency-Saliency Knapsack

MPruner: Optimizing Neural Network Size with CKA-Based Mutual Information Pruning

PruneAug: Bridging DNN Pruning and Inference Latency on Diverse Sparse Platforms Using Automatic Layerwise Block Pruning

Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix

Network Pruning Spaces

Global Sparse Momentum SGD for Pruning Very Deep Neural Networks

HALP: Hardware-Aware Latency Pruning