Abstract:Network pruning and quantization are proven to be effective ways for deep model compression. To obtain a highly compact model, most methods first perform network pruning and then conduct quantization based on the pruned model. However, this strategy may ignore that the pruning and quantization would affect each other and thus performing them separately may lead to sub-optimal performance. To address this, performing pruning and quantization jointly is essential. Nevertheless, how to make a trade-off between pruning and quantization is non-trivial. Moreover, existing compression methods often rely on some pre-defined compression configurations (i.e., pruning rates or bitwidths). Some attempts have been made to search for optimal configurations, which however may take unbearable optimization cost. To address these issues, we devise a simple yet effective method named Single-path Bit Sharing (SBS) for automatic loss-aware model compression. To this end, we consider the network pruning as a special case of quantization and provide a unified view for model pruning and quantization. We then introduce a single-path model to encode all candidate compression configurations, where a high bitwidth value will be decomposed into the sum of a lowest bitwidth value and a series of re-assignment offsets. Relying on the single-path model, we introduce learnable binary gates to encode the choice of configurations and learn the binary gates and model parameters jointly. More importantly, the configuration search problem can be transformed into a subset selection problem, which helps to significantly reduce the optimization difficulty and computation cost. In this way, the compression configurations of each layer and the trade-off between pruning and quantization can be automatically determined. Extensive experiments on CIFAR-100 and ImageNet show that SBS significantly reduces computation cost while achieving promising performance. For example, our SBS compressed MobileNetV2 achieves 22.6× Bit-Operation (BOP) reduction with only 0.1% drop in the Top-1 accuracy.

DSIRBS : A Layer-wise Balanced DNN Weight Pruning Method.

SUBP: Soft Uniform Block Pruning for 1 X N Sparse CNNs Multithreading Acceleration

Learning Slimming SSD Through Pruning and Knowledge Distillation

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

SUBP: Soft Uniform Block Pruning for 1xn Sparse CNNs Multithreading Acceleration

A Feature-map Discriminant Perspective for Pruning Deep Neural Networks

Layerwise Sparse Coding for Pruned Deep Neural Networks with Extreme Compression Ratio.

An efficient pruning and fine-tuning method for deep spiking neural network

DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocation

A systematic DNN weight pruning framework based on symmetric accelerated stochastic ADMM

DARB: A Density-Aware Regular-Block Pruning for Deep Neural Networks

Efficient Network Compression Through Smooth-Lasso Constraint

A Dynamic Pruning Method on Multiple Sparse Structures in Deep Neural Networks

Three-Stage Global Channel Pruning for Resources-Limited Platform

EPSD: Early Pruning with Self-Distillation for Efficient Model Compression

BBS: Bi-directional Bit-level Sparsity for Deep Learning Acceleration

Differential Evolution Based Layer-Wise Weight Pruning for Compressing Deep Neural Networks

Single-path Bit Sharing for Automatic Loss-aware Model Compression

Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon

Surrogate Lagrangian Relaxation: A Path To Retrain-free Deep Neural Network Pruning

ResNet Can Be Pruned 60x: Introducing Network Purification and Unused Path Removal (P-RM) after Weight Pruning