Abstract:Deep learning models have evolved into powerful tools that can be used for many artificial intelligence tasks. However, deploying deep neural networks into real-world applications is still challenging due to their high computational complexity and storage overhead. Fortunately, a densely connected neural network can be converted into a sparsely connected network with low resource demand by the neural network compression. Since deep neural networks are complicated, compression mechanism should find a tradeoff between compression ratio and model accuracy. In this article, by analyzing the statistics of channel connection, we propose an interactive neural network compression mechanism including out-in-channel pruning and neural network quantization. Many channel pruning works apply structured sparsity regularization on each layer separately. We consider correlations between successive layers to retain predictive power of the compact network. A global greedy pruning algorithm is designed to remove redundant out-in-channels in an iterative way. Moreover, in order to solve the shortcomings of the one-shot quantization, we propose the incremental quantization algorithm in the dimension of the output channel, which can smooth network fluctuations and recover accuracy better during retraining. Our mechanism is comprehensively evaluated with various Convolutional Neural Networks (CNN) architectures on popular datasets. Notably, on ImageNet-1K, the out-in-channel pruning reduce 54.0 percent FLOPS on AlexNet and 50.0 percent FLOPs on ResNet-50 with only 0.15 and 0.37 percent top-1 accuracy drop respectively. On classification and style transfer tasks, the superiority of incremental quantization increases with the decrease of the number of quantization bits.

Neural Network Light Weighting Approach Using Multi-Metric Evaluation of Convolution Kernels

Class-Aware Pruning for Efficient Neural Networks

Structured Deep Neural Network Pruning by Varying Regularization Parameters.

A Pruning Method Based on the Dissimilarity of Angle among Channels and Filters

A Unified Framework of DNN Weight Pruning and Weight Clustering/Quantization Using ADMM

Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression

A Systematic DNN Weight Pruning Framework Using Alternating Direction Method of Multipliers

Reweighted Alternating Direction Method of Multipliers for DNN weight pruning

Students and teachers learning together: a robust training strategy for neural network pruning

Multiple Sparse Spaces Network Pruning Via a Joint Similarity Criterion

Complex hybrid weighted pruning method for accelerating convolutional neural networks

Progressive DNN Compression: A Key to Achieve Ultra-High Weight Pruning and Quantization Rates using ADMM

A Clustering Pruning Method Based on Multidimensional Channel Information

Learning Efficient Convolutional Networks Through Network Slimming.

A novel weight pruning strategy for light weight neural networks with application to the diagnosis of skin disease

A Convolutional Neural Network Based on Optimized Structure and Its Lightweighting

Small Contributions, Small Networks: Efficient Neural Network Pruning Based on Relative Importance

Reliable Identification of Redundant Kernels for Convolutional Neural Network Compression.

PCONV: the Missing but Desirable Sparsity in DNN Weight Pruning for Real-Time Execution on Mobile Devices.

Learning Low Resource Consumption CNN through Pruning and Quantization

Intermediate-grained kernel elements pruning with structured sparsity