Abstract:DNN pruning is a popular way to reduce the size of a model, improve the inference latency, and minimize the power consumption on DNN accelerators. However, existing approaches might be too complex, expensive or ineffective to apply to a variety of vision/language tasks, DNN architectures and to honor structured pruning constraints. In this paper, we propose an efficient yet effective train-time pruning scheme, Parameter-free Differentiable Pruning (PDP), which offers state-of-the-art qualities in model size, accuracy, and training cost. PDP uses a dynamic function of weights during training to generate soft pruning masks for the weights in a parameter-free manner for a given pruning target. While differentiable, the simplicity and efficiency of PDP make it universal enough to deliver state-of-the-art random/structured/channel pruning results on various vision and natural language tasks. For example, for MobileNet-v1, PDP can achieve 68.2% top-1 ImageNet1k accuracy at 86.6% sparsity, which is 1.7% higher accuracy than those from the state-of-the-art algorithms. Also, PDP yields over 83.1% accuracy on Multi-Genre Natural Language Inference with 90% sparsity for BERT, while the next best from the existing techniques shows 81.5% accuracy. In addition, PDP can be applied to structured pruning, such as N:M pruning and channel pruning. For 1:4 structured pruning of ResNet18, PDP improved the top-1 ImageNet1k accuracy by over 3.6% over the state-of-the-art. For channel pruning of ResNet50, PDP reduced the top-1 ImageNet1k accuracy by 0.6% from the state-of-the-art.

DMPP: Differentiable Multi-Pruner and Predictor for Neural Network Pruning

Class-Aware Pruning for Efficient Neural Networks

A Feature-map Discriminant Perspective for Pruning Deep Neural Networks

Structured Probabilistic Pruning for Convolutional Neural Network Acceleration.

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

PDP: Parameter-free Differentiable Pruning is All You Need

Progressive Multi-Level Distillation Learning for Pruning Network

DHP: Differentiable Meta Pruning via HyperNetworks

A Dynamic Pruning Method on Multiple Sparse Structures in Deep Neural Networks

Adaptive Search-and-Training for Robust and Efficient Network Pruning

Reweighted Alternating Direction Method of Multipliers for DNN weight pruning

DRIVE: Dual Gradient-Based Rapid Iterative Pruning

Differentiable channel pruning guided via attention mechanism: a novel neural network pruning approach

Global Sparse Momentum SGD for Pruning Very Deep Neural Networks

Non-Parametric Adaptive Network Pruning

Manifold Regularized Dynamic Network Pruning

GenExp: Multi-objective pruning for deep neural network based on genetic algorithm

Joint Multi-Dimension Pruning via Numerical Gradient Update

Differentiable Channel Pruning Search

Separate, Dynamic and Differentiable (SMART) Pruner for Block/Output Channel Pruning on Computer Vision Tasks

A Systematic DNN Weight Pruning Framework Using Alternating Direction Method of Multipliers