Abstract:Compressing Deep Neural Network (DNN) models to alleviate the storage and computation requirements is essential for practical applications, especially for resource limited devices. Although capable of reducing a reasonable amount of model parameters, previous unstructured or structured weight pruning methods can hardly truly accelerate inference, either due to the poor hardware compatibility of the unstructured sparsity or due to the low sparse rate of the structurally pruned network. Aiming at reducing both storage and computation, as well as preserving the original task performance, we propose a generalized weight unification framework at a hardware compatible micro-structured level to achieve high amount of compression and acceleration. Weight coefficients of a selected micro-structured block are unified to reduce the storage and computation of the block without changing the neuron connections, which turns to a micro-structured pruning special case when all unified coefficients are set to zero, where neuron connections (hence storage and computation) are completely removed. In addition, we developed an effective training framework based on the alternating direction method of multipliers (ADMM), which converts our complex constrained optimization into separately solvable subproblems. Through iteratively optimizing the subproblems, the desired micro-structure can be ensured with high compression ratio and low performance degradation. We extensively evaluated our method using a variety of benchmark models and datasets for different applications. Experimental results demonstrate state-of-the-art performance.

Compressing Neural Networks By Applying Frequent Item-Set Mining

Class-Aware Pruning for Efficient Neural Networks

Pruning by Training: A Novel Deep Neural Network Compression Framework for Image Processing.

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding

Filter Pruning Via Feature Map Clustering.

A Dynamic Pruning Method on Multiple Sparse Structures in Deep Neural Networks

An efficient pruning and fine-tuning method for deep spiking neural network

Quantisation and Pruning for Neural Network Compression and Regularisation

Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression

Compressing Deep Neural Network for Facial Landmarks Detection

Pruning and quantization for deep neural network acceleration: A survey

Model Compression for Deep Neural Networks: A Survey

On Compression of Unsupervised Neural Nets by Pruning Weak Connections

Pruning at a Glance: Global Neural Pruning for Model Compression

Identifying and Pruning Redundant Structures for Deep Neural Networks

Pruning the Deep Neural Network by Similar Function

Anonymous Model Pruning for Compressing Deep Neural Networks

What Do Compressed Deep Neural Networks Forget?

Minimally Invasive Surgery for Sparse Neural Networks in Contrastive Manner

AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates