Abstract:Pruning neural networks has become popular in the last decade when it was shown that a large number of weights can be safely removed from modern neural networks without compromising accuracy. Numerous pruning methods have been proposed since then, each claiming to be better than the previous. Many state-of-the-art (SOTA) techniques today rely on complex pruning methodologies utilizing importance scores, getting feedback through back-propagation or having heuristics-based pruning rules amongst others. We question this pattern of introducing complexity in order to achieve better pruning results. We benchmark these SOTA techniques against Global Magnitude Pruning (Global MP), a naive pruning baseline, to evaluate whether complexity is really needed to achieve higher performance. Global MP ranks weights in order of their magnitudes and prunes the smallest ones. Hence, in its vanilla form, it is one of the simplest pruning techniques. Surprisingly, we find that vanilla Global MP outperforms all the other SOTA techniques and achieves a new SOTA result. It also achieves good performance on FLOPs sparsification, which we find is enhanced, when pruning is conducted in a gradual fashion. We also find that Global MP is generalizable across tasks, datasets and models with superior performance. Moreover, a common issue that many pruning algorithms run into at high sparsity rates, namely, layer-collapse, can be easily fixed in Global MP by setting a minimum threshold of weights to be retained in each layer. Lastly, unlike many other SOTA techniques, Global MP does not require any additional algorithm specific hyper-parameters and is very straightforward to tune and implement. We showcase our findings on various models (WRN-28-8, ResNet-32, ResNet-50, MobileNet-V1 and FastGRNN) and multiple datasets (CIFAR-10, ImageNet and HAR-2). Code is available at https://github.com/manasgupta-1/GlobalMP.

Stage-Wise Magnitude-Based Pruning for Recurrent Neural Networks

Structured Pruning of Recurrent Neural Networks through Neuron Selection

Class-Aware Pruning for Efficient Neural Networks

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

Structured Probabilistic Pruning for Convolutional Neural Network Acceleration.

Structured Deep Neural Network Pruning by Varying Regularization Parameters.

Optimization based Layer-wise Magnitude-based Pruning for DNN Compression

Where to Prune: Using LSTM to Guide End-to-end Pruning

Efficient DNN Neuron Pruning by Minimizing Layer-wise Nonlinear Reconstruction Error

Pruning of Deep Spiking Neural Networks Through Gradient Rewiring.

Network Compression Via Recursive Bayesian Pruning.

Runtime Neural Pruning

Where to Prune: Using LSTM to Guide Data-Dependent Soft Pruning

Is Complexity Required for Neural Network Pruning? A Case Study on Global Magnitude Pruning

Three-Stage Global Channel Pruning for Resources-Limited Platform

Rapid Structural Pruning of Neural Networks with Set-based Task-Adaptive Meta-Pruning

Structured Pruning for Multi-Task Deep Neural Networks

Sparse optimization guided pruning for neural networks

Automatic Mapping of the Best-Suited DNN Pruning Schemes for Real-Time Mobile Acceleration

RGP: Neural Network Pruning Through Regular Graph With Edges Swapping