Abstract:Pruning neural networks has become popular in the last decade when it was shown that a large number of weights can be safely removed from modern neural networks without compromising accuracy. Numerous pruning methods have been proposed since, each claiming to be better than prior art, however, at the cost of increasingly complex pruning methodologies. These methodologies include utilizing importance scores, getting feedback through back-propagation or having heuristics-based pruning rules amongst others. In this work, we question whether this pattern of introducing complexity is really necessary to achieve better pruning results. We benchmark these SOTA techniques against a simple pruning baseline, namely, Global Magnitude Pruning (Global MP), that ranks weights in order of their magnitudes and prunes the smallest ones. Surprisingly, we find that vanilla Global MP performs very well against the SOTA techniques. When considering sparsity-accuracy trade-off, Global MP performs better than all SOTA techniques at all sparsity ratios. When considering FLOPs-accuracy trade-off, some SOTA techniques outperform Global MP at lower sparsity ratios, however, Global MP starts performing well at high sparsity ratios and performs very well at extremely high sparsity ratios. Moreover, we find that a common issue that many pruning algorithms run into at high sparsity rates, namely, layer-collapse, can be easily fixed in Global MP. We explore why layer collapse occurs in networks and how it can be mitigated in Global MP by utilizing a technique called Minimum Threshold. We showcase the above findings on various models (WRN-28-8, ResNet-32, ResNet-50, MobileNet-V1 and FastGRNN) and multiple datasets (CIFAR-10, ImageNet and HAR-2). Code is available at <a class="link-external link-https" href="https://github.com/manasgupta-1/GlobalMP" rel="external noopener nofollow">this https URL</a>.

An algorithm for pruning redundant modules in min-max modular network with GZC function

Typical sample selection and redundancy reduction for min-max modular network with GZC function

A Min-Max Modular Network with Gaussian-Zero-Crossing Function

An Algorithm for Pruning Redundant Modules in Min-Ma Modular Network

Structure pruning strategies for min-max modular network

Class-Aware Pruning for Efficient Neural Networks

MaxQ: Multi-Axis Query for N:M Sparsity Network

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

Structured Pruning for Efficient Convolutional Neural Networks Via Incremental Regularization

A Gaussian Zero-Crossing Discriminant Function For Min-Max Modular Neural Networks

Is Complexity Required for Neural Network Pruning? A Case Study on Global Magnitude Pruning

Module Combination Based on Decision Tree in Min-Max Modular Network

A Pruning Method Based on the Dissimilarity of Angle among Channels and Filters

A Gaussian Zero Crossing Discriminant Function

Towards Thinner Convolutional Neural Networks Through Gradually Global Pruning

Multi-domain clustering pruning: Exploring space and frequency similarity based on GAN

A New Supervised Clustering Algorithm Based on Min-Max Modular Network with Gaussian-Zero-Crossing Functions

Multi-Dimensional Dynamic Pruning: Exploring Spatial and Channel Fuzzy Sparsity

Filter Pruning for CNN with Enhanced Linear Representation Redundancy

RGP: Neural Network Pruning Through Regular Graph With Edges Swapping