Abstract:Distributed synchronous stochastic gradient descent (SGD) algorithms are widely used in large-scale deep learning applications, while it is known that the communication bottleneck limits the scalability of the distributed system. Gradient sparsification is a promising technique to significantly reduce the communication traffic, while pipelining can further overlap the communications with computations. However, gradient sparsification introduces extra computation time, and pipelining requires many layer-wise communications which introduce significant communication startup overheads. Merging gradients from neighbor layers could reduce the startup overheads, but on the other hand it would increase the computation time of sparsification and the waiting time for the gradient computation. In this paper, we formulate the trade-off between communications and computations (including backward computation and gradient sparsification) as an optimization problem, and derive an optimal solution to the problem. We further develop the optimal merged gradient sparsification algorithm with SGD (OMGS-SGD) for distributed training of deep learning. We conduct extensive experiments to verify the convergence properties and scaling performance of OMGS-SGD. Experimental results show that OMGS-SGD achieves up to 31% end-to-end time efficiency improvement over the state-of-the-art sparsified SGD while preserving nearly consistent convergence performance with original SGD without sparsification on a 16-GPU cluster connected with 1Gbps Ethernet.

Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient

Communication-Adaptive Stochastic Gradient Methods for Distributed Learning

Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees.

GradSA: Gradient Sparsification and Accumulation for Communication-Efficient Distributed Deep Learning

Communication-Efficient Distributed Stochastic Gradient Descent with Pooling Operator

Toward Communication Efficient Adaptive Gradient Method

Sparse Communication for Training Deep Networks

Adaptive Stochastic Gradient Descent for Fast and Communication-Efficient Distributed Learning

Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimization

Sparse Gradient Compression For Distributed Sgd

LASG: Lazily Aggregated Stochastic Gradients for Communication-Efficient Distributed Learning

Gradient Sparsification for Communication-Efficient Distributed Optimization

Communication-Censored Distributed Stochastic Gradient Descent

Communication-Efficient Distributed Deep Learning with Merged Gradient Sparsification on GPUs

Efficient Distributed Stochastic Gradient Descent Through Gaussian Averaging.

Communication-Constrained Distributed Learning: TSI-Aided Asynchronous Optimization with Stale Gradient

A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication.

Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods

GSASG: Global Sparsification with Adaptive Aggregated Stochastic Gradients for Communication-Efficient Federated Learning

Communication-Efficient and Byzantine-Robust Distributed Stochastic Learning with Arbitrary Number of Corrupted Workers