Abstract:Many model compression techniques of Deep Neural Networks (DNNs) have been investigated, including weight pruning, weight clustering and quantization, etc. Weight pruning leverages the redundancy in the number of weights in DNNs, while weight clustering/quantization leverages the redundancy in the number of bit representations of weights. They can be effectively combined in order to exploit the maximum degree of redundancy. However, there lacks a systematic investigation in literature towards this direction. In this paper, we fill this void and develop a unified, systematic framework of DNN weight pruning and clustering/quantization using Alternating Direction Method of Multipliers (ADMM), a powerful technique in optimization theory to deal with non-convex optimization problems. Both DNN weight pruning and clustering/quantization, as well as their combinations, can be solved in a unified manner. For further performance improvement in this framework, we adopt multiple techniques including iterative weight quantization and retraining, joint weight clustering training and centroid updating, weight clustering retraining, etc. The proposed framework achieves significant improvements both in individual weight pruning and clustering/quantization problems, as well as their combinations. For weight pruning alone, we achieve 167x weight reduction in LeNet-5, 24.7x in AlexNet, and 23.4x in VGGNet, without any accuracy loss. For the combination of DNN weight pruning and clustering/quantization, we achieve 1,910x and 210x storage reduction of weight data on LeNet-5 and AlexNet, respectively, without accuracy loss. Our codes and models are released at the link http://bit.ly/2D3F0np

WeightGrad: Geo-Distributed Data Analysis Using Quantization for Faster Convergence and Better Accuracy

Distributed Subgradient Method with Random Quantization and Flexible Weights: Convergence Analysis.

Log-Scale Quantization in Distributed First-Order Methods: Gradient-based Learning from Distributed Data

Improved Quantization Strategies for Managing Heavy-tailed Gradients in Distributed Learning

Secure Federated Learning with Model Compression.

Lazily Aggregated Quantized Gradient Innovation for Communication-Efficient Federated Learning.

A Unified Framework of DNN Weight Pruning and Weight Clustering/Quantization Using ADMM

Near-Lossless Gradient Compression for Data-Parallel Distributed DNN Training

Convergence Theory of Generalized Distributed Subgradient Method with Random Quantization

Global-QSGD: Practical Floatless Quantization for Distributed Learning with Theoretical Guarantees

Hierarchical Weight Averaging for Deep Neural Networks

Distributed Constrained Optimization With Delayed Subgradient Information Over Time-Varying Network Under Adaptive Quantization

Progressive DNN Compression: A Key to Achieve Ultra-High Weight Pruning and Quantization Rates using ADMM

Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization

DQSGD: DYNAMIC QUANTIZED STOCHASTIC GRADIENT DESCENT FOR COMMUNICATION-EFFICIENT DISTRIBUTED LEARNING

An Efficient Bandwidth-Adaptive Gradient Compression Algorithm for Distributed Training of Deep Neural Networks

Quantized Adam with Error Feedback

Weighted Aggregating Stochastic Gradient Descent for Parallel Deep Learning

Toward Extremely Low Bit and Lossless Accuracy in DNNs with Progressive ADMM

Evaluation and Optimization of Gradient Compression for Distributed Deep Learning