Abstract:When using stochastic gradient descent (SGD) to solve large-scale machine learning problems especially deep learning problems, a common practice of data processing is to shuffle the training data, partition the data across multiple threads/machines if needed, and then perform several epochs of training on the re-shuffled (either locally or globally) data. The above procedure makes the instances used to compute the gradients no longer independently sampled from the training data set, which contradicts with the basic assumptions of conventional convergence analysis of SGD. Then does the distributed SGD method have desirable convergence properties in this practical situation? In this paper, we give answers to this question. First, we give a mathematical formulation for the practical data processing procedure in distributed machine learning, which we call (data partition with) global/local shuffling. We observe that global shuffling is equivalent to without-replacement sampling if the shuffling operations are independent. Second, we prove SGD with global shuffling and local shuffling has convergence guarantee for non-convex tasks like deep learning. The convergence rate for local shuffling is slower than that for global shuffling, since it will lose some information if there’s no communication between partitioned data. We also consider the situation when the permutation after shuffling is not uniformly distributed (We call it insufficient shuffling), and discuss the condition under which this insufficiency will not influence the convergence rate. Finally, we give the convergence analysis in convex case. An interesting finding is that, the non-convex tasks like deep learning are more suitable to apply shuffling comparing to the convex tasks. Our theoretical results provide important insights to large-scale machine learning, especially in the selection of data processing methods in order to achieve faster convergence and good speedup. Our theoretical findings are verified by extensive experiments on logistic regression and deep neural networks.

Accelerating local SGD for non-IID data using variance reduction

Variance Reduced Local SGD with Lower Communication Complexity

Accelerating SGD Using Flexible Variance Reduction on Large-Scale Datasets

Fvr-Sgd: A New Flexible Variance-Reduction Method For Sgd On Large-Scale Datasets

Variance Reduced Diffusion Adaptation for Online Learning over Networks

Distributed Gradient Descent with Many Local Steps in Overparameterized Models

DisSAGD: A Distributed Parameter Update Scheme Based on Variance Reduction

Aperiodic Local SGD: Beyond Local SGD.

Convergence Analysis of Distributed Stochastic Gradient Descent with Shuffling

STL-SGD: Speeding Up Local SGD with Stagewise Communication Period

Lock-Free Parallelization for Variance-Reduced Stochastic Gradient Descent on Streaming Data

On the Convergence of Communication-Efficient Local SGD for Federated Learning

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

Variance-reduced Reshuffling Gradient Descent for Nonconvex Optimization: Centralized and Distributed Algorithms

Fast Asynchronous Parallel Stochastic Gradient Decent

Asynchronous Local-SGD Training for Language Modeling

Decentralized Local Updates with Dual-Slow Estimation and Momentum-Based Variance-Reduction for Non-Convex Optimization

On the Convergence of Local Descent Methods in Federated Learning

FLSGD: Free Local SGD with Parallel Synchronization

Convergence of Distributed Stochastic Variance Reduced Methods Without Sampling Extra Data

Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods