Abstract:Deep learning models have evolved into powerful tools that can be used for many artificial intelligence tasks. However, deploying deep neural networks into real-world applications is still challenging due to their high computational complexity and storage overhead. Fortunately, a densely connected neural network can be converted into a sparsely connected network with low resource demand by the neural network compression. Since deep neural networks are complicated, compression mechanism should find a tradeoff between compression ratio and model accuracy. In this article, by analyzing the statistics of channel connection, we propose an interactive neural network compression mechanism including out-in-channel pruning and neural network quantization. Many channel pruning works apply structured sparsity regularization on each layer separately. We consider correlations between successive layers to retain predictive power of the compact network. A global greedy pruning algorithm is designed to remove redundant out-in-channels in an iterative way. Moreover, in order to solve the shortcomings of the one-shot quantization, we propose the incremental quantization algorithm in the dimension of the output channel, which can smooth network fluctuations and recover accuracy better during retraining. Our mechanism is comprehensively evaluated with various Convolutional Neural Networks (CNN) architectures on popular datasets. Notably, on ImageNet-1K, the out-in-channel pruning reduce 54.0 percent FLOPS on AlexNet and 50.0 percent FLOPs on ResNet-50 with only 0.15 and 0.37 percent top-1 accuracy drop respectively. On classification and style transfer tasks, the superiority of incremental quantization increases with the decrease of the number of quantization bits.

Transfer Knowledge for High Sparsity in Deep Neural Networks

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

Class-Aware Pruning for Efficient Neural Networks

Pruning by Training: A Novel Deep Neural Network Compression Framework for Image Processing.

Students and teachers learning together: a robust training strategy for neural network pruning

A Model Compression Method Using Significant Data and Knowledge Distillation

Efficient Network Compression Through Smooth-Lasso Constraint

Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

Knowledge from the Original Network: Restore a Better Pruned Network with Knowledge Distillation

Using Distillation to Improve Network Performance after Pruning and Quantization

Compressing Deep Neural Networks With Sparse Matrix Factorization

Few Sample Knowledge Distillation for Efficient Network Compression

PrUE: Distilling Knowledge from Sparse Teacher Networks

A deep neural network compression algorithm based on knowledge transfer for edge devices

An efficient pruning and fine-tuning method for deep spiking neural network

Overview of Deep Convolutional Neural Network Pruning

Learning Instance-wise Sparsity for Accelerating Deep Models

Accurate Neural Network Pruning Requires Rethinking Sparse Optimization

Learning Low Resource Consumption CNN through Pruning and Quantization

Multiple Sparse Spaces Network Pruning Via a Joint Similarity Criterion

Sparse Deep Transfer Learning for Convolutional Neural Network