Abstract:In recent years, deep neural networks (DNN) have attracted increasing attention because of their excellent performance in computer vision and natural language processing. The success of deep learning is due to the fact that the models have more layers and more parameters, which gives them stronger nonlinear fitting ability. Furthermore, the continuous updating of hardware equipment makes it possible to quickly train deep learning models. The development of deep learning is driven by the greater amounts of available annotated or unannotated data. Specifically, large-scale data provide models with greater learning space and stronger generalization ability. Although the performance of deep neural networks is significant, they are difficult to deploy in embedded or mobile devices with limited hardware due to their large number of parameters and high storage and computing costs. Recent studies have found that deep models based on a convolutional neural network are characterized by parameter redundancy as well as parameters that are irrelevant to the final model results, which provides theoretical support for the compression of deep network models. Therefore, determining ways to reduce model size while retaining model precision has become a hot research issue. Model compression refers to the reduction of a trained model through some operation to obtain a lightweight network with equivalent performance. After model compression, there are fewer network parameters and usually a reduction in the computation required, which greatly reduces the computational and storage costs and enables the deployment of the model in restricted hardware conditions. In this paper, the achievements and progress made in recent years by domestic and foreign scholars with respect to model compressionwere classified and summarized and their advantages and disadvantages were evaluated, including network pruning, parameter sharing, quantization, network decomposition, and network distillation. Then, existing problems and the future development of model compression were discussed.

Re-training and parameter sharing with the Hash trick for compressing convolutional neural networks

Regularized Training Framework for Combining Pruning and Quantization to Compress Neural Networks

A Model Compression Method Using Significant Data and Knowledge Distillation

A Highly Efficient Training-Aware Convolutional Neural Network Compression Paradigm

Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding

A Survey of Model Compression and Acceleration for Deep Neural Networks.

Layer-Wise Training To Create Efficient Convolutional Neural Networks

Compressing complex convolutional neural network based on an improved deep compression algorithm

Efficient Neural Network Compression Inspired by Compressive Sensing.

Towards Convolutional Neural Networks Compression Via Global Error Reconstruction.

A Low Bit-Width Parameter Representation Method for Hardware-Oriented Convolution Neural Networks.

Efficient Deep Convolutional Neural Networks Accelerator Without Multiplication and Retraining

A Survey of Model Compression for Deep Neural Networks

SCWC: Structured channel weight sharing to compress convolutional neural networks

Bit-Quantized-Net: an Effective Method for Compressing Deep Neural Networks.

A New Compression Method for Deep Neural Networks with Accuracy Improvement

Holistic CNN Compression Via Low-Rank Decomposition with Knowledge Transfer.

Drastically Reducing the Number of Trainable Parameters in Deep CNNs by Inter-layer Kernel-sharing

A Unified Approximation Framework for Compressing and Accelerating Deep Neural Networks

Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression

TEC-CNN: Towards Efficient Compressing Convolutional Neural Nets with Low-rank Tensor Decomposition