Abstract:A standard approach in large scale machine learning is distributed stochastic gradient training, which requires the computation of aggregated stochastic gradients over multiple nodes on a network. Communication is a major bottleneck in such applications, and in recent years, compressed stochastic gradient methods such as QSGD (quantized SGD) and sparse SGD have been proposed to reduce communication. It was also shown that error compensation can be combined with compression to achieve better convergence in a scheme that each node compresses its local stochastic gradient and broadcast the result to all other nodes over the network in a single pass. However, such a single pass broadcast approach is not realistic in many practical implementations. For example, under the popular parameter server model for distributed learning, the worker nodes need to send the compressed local gradients to the parameter server, which performs the aggregation. The parameter server has to compress the aggregated stochastic gradient again before sending it back to the worker nodes. In this work, we provide a detailed analysis on this two-pass communication model and its asynchronous parallel variant, with error-compensated compression both on the worker nodes and on the parameter server. We show that the error-compensated stochastic gradient algorithm admits three very nice properties: 1) it is compatible with an \emph{arbitrary} compression technique; 2) it admits an improved convergence rate than the non error-compensated stochastic gradient methods such as QSGD and sparse SGD; 3) it admits linear speedup with respect to the number of workers. The empirical study is also conducted to validate our theoretical results.

Communication-Efficient Distributed Deep Learning with Merged Gradient Sparsification on GPUs

Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient

MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning

Sparse Communication for Training Deep Networks

Sparse Gradient Compression For Distributed Sgd

Evaluation and Optimization of Gradient Compression for Distributed Deep Learning

SparDL: Distributed Deep Learning Training with Efficient Sparse Communication

A Distributed SGD Algorithm with Global Sketching for Deep Learning Training Acceleration

Gradient Sparsification for Communication-Efficient Distributed Optimization

Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent

Detached Error Feedback for Distributed SGD with Random Sparsification

Preserving Near-Optimal Gradient Sparsification Cost for Scalable Distributed Deep Learning

DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression

S2 Reducer: High-Performance Sparse Communication to Accelerate Distributed Deep Learning

CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation

Dynamic layer-wise sparsification for distributed deep learning

SAP-SGD: Accelerating Distributed Parallel Training with High Communication Efficiency on Heterogeneous Clusters

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Adaptive Consensus Gradients Aggregation for Scaled Distributed Training

DaSGD: Squeezing SGD Parallelization Performance in Distributed Training Using Delayed Averaging

ABS-SGD: A Delayed Synchronous Stochastic Gradient Descent Algorithm with Adaptive Batch Size for Heterogeneous GPU Clusters.