Abstract:Deep neural network (DNN) obtained satisfactory results on different vision tasks; however, they usually suffer from large models and massive parameters during model deployment. While DNN compression can reduce the memory footprint of deep model effectively, so that the deep model can be deployed on portable devices. However, most of the existing model compression methods cost lots of time, e.g., vector quantization or pruning, which makes them inept to the application that needs fast computation. In this paper, we therefore explore how to accelerate the model compression process by reducing the computation cost. Then, we propose a new model compression method, termed dictionary-pair-based fast data-free DNN compression, which aims at reducing the memory consumption of DNNs without extra training and can greatly improve the compression efficiency. Specifically, our method performs tensor decomposition of DNN model with a fast dictionary-pair learning-based reconstruction approach, which can be deployed on different weight layers (e.g., convolution and fully connected layers). Given a pre-trained DNN model, we first divide the parameters (i.e., weights) of each layer into a series of partitions for dictionary pair-driven fast reconstruction, which can potentially discover more fine-grained information and provide the possibility for parallel model compression. Then, dictionaries of less memory occupation are learned to reconstruct the weights. Moreover, automatic hyper-parameter tuning and shared-dictionary mechanism is proposed to improve the model performance and availability. Extensive experiments on popular DNN models (i.e., VGG-16, ResNet-18 and ResNet-50) showed that our proposed weight compression method can significantly reduce the memory footprint and speed up the compression process, with less performance loss.

Improved Model Compression Method Based on Information Entropy

Model Compression for Deep Neural Networks: A Survey

A Model Compression Method Using Significant Data and Knowledge Distillation

Deep Learning Model Compression Techniques: Advances, Opportunities, and Perspective

Compressing complex convolutional neural network based on an improved deep compression algorithm

Deep learning model compression using network sensitivity and gradients

Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding

Kernel-wise difference minimization for convolutional neural network compression in metaverse

Dictionary Pair-based Data-Free Fast Deep Neural Network Compression

A Novel Deep Learning Model Compression Algorithm

Adaptive Layerwise Quantization for Deep Neural Network Compression

Weightless: Lossy Weight Encoding For Deep Neural Network Compression

On Compressing Deep Models by Low Rank and Sparse Decomposition.

Improved deep learning image compression model: performance optimization based on convolutional modules and local attention mechanism

Fast data-free model compression via dictionary-pair reconstruction

A Survey of Model Compression and Acceleration for Deep Neural Networks.

On Efficient Neural Network Architectures for Image Compression

To Compress, or Not to Compress: Characterizing Deep Learning Model Compression for Embedded Inference

Neural Network Compression Via Sparse Optimization

Lossy and Lossless (L$^2$) Post-training Model Size Compression

Analysis of Model Compression Using Knowledge Distillation