Abstract:When using stochastic gradient descent (SGD) to solve large-scale machine learning problems especially deep learning problems, a common practice of data processing is to shuffle the training data, partition the data across multiple threads/machines if needed, and then perform several epochs of training on the re-shuffled (either locally or globally) data. The above procedure makes the instances used to compute the gradients no longer independently sampled from the training data set, which contradicts with the basic assumptions of conventional convergence analysis of SGD. Then does the distributed SGD method have desirable convergence properties in this practical situation? In this paper, we give answers to this question. First, we give a mathematical formulation for the practical data processing procedure in distributed machine learning, which we call (data partition with) global/local shuffling. We observe that global shuffling is equivalent to without-replacement sampling if the shuffling operations are independent. Second, we prove SGD with global shuffling and local shuffling has convergence guarantee for non-convex tasks like deep learning. The convergence rate for local shuffling is slower than that for global shuffling, since it will lose some information if there’s no communication between partitioned data. We also consider the situation when the permutation after shuffling is not uniformly distributed (We call it insufficient shuffling), and discuss the condition under which this insufficiency will not influence the convergence rate. Finally, we give the convergence analysis in convex case. An interesting finding is that, the non-convex tasks like deep learning are more suitable to apply shuffling comparing to the convex tasks. Our theoretical results provide important insights to large-scale machine learning, especially in the selection of data processing methods in order to achieve faster convergence and good speedup. Our theoretical findings are verified by extensive experiments on logistic regression and deep neural networks.

Convergence Theory of Generalized Distributed Subgradient Method with Random Quantization

Distributed Subgradient Method with Random Quantization and Flexible Weights: Convergence Analysis.

Distributed Constrained Optimization With Delayed Subgradient Information Over Time-Varying Network Under Adaptive Quantization

Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization

Truncated Non-Uniform Quantization for Distributed SGD

DQ-SGD: Dynamic Quantization in SGD for Communication-Efficient Distributed Learning

Distributed Optimization with Quantized Gradient Descent

Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms

Improved Quantization Strategies for Managing Heavy-tailed Gradients in Distributed Learning

DQSGD: DYNAMIC QUANTIZED STOCHASTIC GRADIENT DESCENT FOR COMMUNICATION-EFFICIENT DISTRIBUTED LEARNING

Distributed Online Quantile Regression over Networks with Quantized Communication.

Quantized Distributed Gradient Tracking Algorithm with Linear Convergence in Directed Networks

Quantized Epoch-SGD for Communication-Efficient Distributed Learning.

Distributed Online Optimization Under Dynamic Adaptive Quantization

A Distributed Training Algorithm of Generative Adversarial Networks with Quantized Gradients

Quantization Avoids Saddle Points in Distributed Optimization

Distributed Zeroth-Order Optimization: Convergence Rates That Match Centralized Counterpart

vqSGD: Vector Quantized Stochastic Gradient Descent

Global-QSGD: Practical Floatless Quantization for Distributed Learning with Theoretical Guarantees

Convergence Analysis of Distributed Stochastic Gradient Descent with Shuffling

Killing Two Birds with One Stone: Quantization Achieves Privacy in Distributed Learning