Abstract:As deep neural networks are growing in size and being increasingly deployed to more resource-limited devices, there has been a recent surge of interest in network pruning methods, which aim to remove less important weights or activations of a given network. A common limitation of most existing pruning techniques, is that they require pre-training of the network at least once before pruning, and thus we can benefit from reduction in memory and computation only at the inference time. However, reducing the training cost of neural networks with rapid structural pruning may be beneficial either to minimize monetary cost with cloud computing or to enable on-device learning on a resource-limited device. Recently introduced random-weight pruning approaches can eliminate the needs of pretraining, but they often obtain suboptimal performance over conventional pruning techniques and also does not allow for faster training since they perform unstructured pruning. To overcome their limitations, we propose Set-based Task-Adaptive Meta Pruning (STAMP), which task-adaptively prunes a network pretrained on a large reference dataset by generating a pruning mask on it as a function of the target dataset. To ensure maximum performance improvements on the target task, we meta-learn the mask generator over different subsets of the reference dataset, such that it can generalize well to any unseen datasets within a few gradient steps of training. We validate STAMP against recent advanced pruning methods on benchmark datasets, on which it not only obtains significantly improved compression rates over the baselines at similar accuracy, but also orders of magnitude faster training speed.

Robust Neural Pruning with Gradient Sampling Optimization for Residual Neural Networks

Class-Aware Pruning for Efficient Neural Networks

Pruning by Training: A Novel Deep Neural Network Compression Framework for Image Processing.

StochGradAdam: Accelerating Neural Networks Training with Stochastic Gradient Sampling

Sparse optimization guided pruning for neural networks

Neural Network Pruning by Gradient Descent

Adaptive Search-and-Training for Robust and Efficient Network Pruning

Pruning of Deep Spiking Neural Networks Through Gradient Rewiring.

Robust Neural Network Pruning by Cooperative Coevolution

Pruning Convolutional Neural Networks via Stochastic Gradient Hard Thresholding

Sparse Training via Boosting Pruning Plasticity with Neuroregeneration

Joint Multi-Dimension Pruning via Numerical Gradient Update

CCSRP: Robust Pruning of Spiking Neural Networks through Cooperative Coevolution

FGGP: Fixed-Rate Gradient-First Gradual Pruning

Stochastic Subnetwork Annealing: A Regularization Technique for Fine Tuning Pruned Subnetworks

Global Sparse Momentum SGD for Pruning Very Deep Neural Networks

Enabling Retrain-free Deep Neural Network Pruning using Surrogate Lagrangian Relaxation

Rapid Structural Pruning of Neural Networks with Set-based Task-Adaptive Meta-Pruning

Beyond Pruning Criteria: The Dominant Role of Fine-Tuning and Adaptive Ratios in Neural Network Robustness

Accurate Neural Network Pruning Requires Rethinking Sparse Optimization

Pruning Deep Convolutional Neural Networks via Gradient Support Pursuit