Abstract:Distributed machine learning (ML) has been extensively studied to meet the explosive growth of training data. A wide range of machine learning models are trained by a family of first-order optimization algorithms, i.e., stochastic gradient descent (SGD). The core operation of SGD is the calculation of gradients. When executing SGD in a distributed environment, the workers need to exchange local gradients through the network. In order to reduce the communication cost, a category of quantification-based compression algorithms are used to transform the gradients to binary format, at the expense of a low precision loss. Although the existing approaches work fine for dense gradients, we find that these methods are ill-suited for many cases where the gradients are sparse and nonuniformly distributed. In this paper, we study is there a compression framework that can efficiently handle sparse and nonuniform gradients? We propose a general compression framework, called SKCompress, to compress both gradient values and gradient keys in sparse gradients. Our first contribution is a sketch-based method that compresses the gradient values. Sketch is a class of algorithm that approximates the distribution of a data stream with a probabilistic data structure. We first use a quantile sketch to generate splits, sort gradient values into buckets, and encode them with the bucket indexes. Our second contribution is a new sketch algorithm, namely MinMaxSketch, which compresses the bucket indexes. MinMaxSketch builds a set of hash tables and solves hash collisions with a MinMax strategy. Since the bucket indexes are nonuniform, we further adopt Huffman coding to compress MinMaxSketch. To compress the keys of sparse gradients, the third contribution of this paper is a delta-binary encoding method that calculates the increment of the gradient keys and encode them with binary format. An adaptive prefix is proposed to assign different sizes to different gradient keys, so that we can save more space. We also theoretically discuss the correctness and the error bound of our proposed methods. To the best of our knowledge, this is the first effort utilizing data sketch to compress gradients in ML. We implement a prototype system in a real cluster of our industrial partner Tencent Inc. and show that our method is up to $$12\times $$ faster than the existing methods.

CASQ: Accelerate Distributed Deep Learning with Sketch-Based Gradient Quantization

Accelerate Distributed Deep Learning with Cluster-Aware Sketch Quantization

Optimizing Quantized Neural Networks in a Weak Curvature Manifold

DQSGD: DYNAMIC QUANTIZED STOCHASTIC GRADIENT DESCENT FOR COMMUNICATION-EFFICIENT DISTRIBUTED LEARNING

DQ-SGD: Dynamic Quantization in SGD for Communication-Efficient Distributed Learning

SK-Gradient: Efficient Communication for Distributed Machine Learning with Data Sketch.

Quantized Epoch-SGD for Communication-Efficient Distributed Learning.

A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication.

JointSQ: Joint Sparsification-Quantization for Distributed Learning

A Distributed SGD Algorithm with Global Sketching for Deep Learning Training Acceleration

Improved Quantization Strategies for Managing Heavy-tailed Gradients in Distributed Learning

Quantized Adaptive Subgradient Algorithms and Their Applications

SKCompress: Compressing Sparse and Nonuniform Gradient in Distributed Machine Learning

Truncated Non-Uniform Quantization for Distributed SGD

GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Training.

Global-QSGD: Practical Floatless Quantization for Distributed Learning with Theoretical Guarantees

SQuant: On-the-Fly Data-Free Quantization Via Diagonal Hessian Approximation

Error compensated quantized SGD and its applications to large-scale distributed optimization

Convergence Theory of Generalized Distributed Subgradient Method with Random Quantization

Stochastic Quantization for Learning Accurate Low-Bit Deep Neural Networks