Abstract:When using stochastic gradient descent (SGD) to solve large-scale machine learning problems especially deep learning problems, a common practice of data processing is to shuffle the training data, partition the data across multiple threads/machines if needed, and then perform several epochs of training on the re-shuffled (either locally or globally) data. The above procedure makes the instances used to compute the gradients no longer independently sampled from the training data set, which contradicts with the basic assumptions of conventional convergence analysis of SGD. Then does the distributed SGD method have desirable convergence properties in this practical situation? In this paper, we give answers to this question. First, we give a mathematical formulation for the practical data processing procedure in distributed machine learning, which we call (data partition with) global/local shuffling. We observe that global shuffling is equivalent to without-replacement sampling if the shuffling operations are independent. Second, we prove SGD with global shuffling and local shuffling has convergence guarantee for non-convex tasks like deep learning. The convergence rate for local shuffling is slower than that for global shuffling, since it will lose some information if there’s no communication between partitioned data. We also consider the situation when the permutation after shuffling is not uniformly distributed (We call it insufficient shuffling), and discuss the condition under which this insufficiency will not influence the convergence rate. Finally, we give the convergence analysis in convex case. An interesting finding is that, the non-convex tasks like deep learning are more suitable to apply shuffling comparing to the convex tasks. Our theoretical results provide important insights to large-scale machine learning, especially in the selection of data processing methods in order to achieve faster convergence and good speedup. Our theoretical findings are verified by extensive experiments on logistic regression and deep neural networks.

Shuffle Private Decentralized Convex Optimization

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

A(DP)$^2$2SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Decentralized Stochastic Optimization with Inherent Privacy Protection

Differentially Private Block-wise Gradient Shuffle for Deep Learning

Privacy-Preserving Push-Pull Method for Decentralized Optimization via State Decomposition

Convergence Analysis of Distributed Stochastic Gradient Descent with Shuffling

Faster Algorithms for User-Level Private Stochastic Convex Optimization

Privacy Preservation in Distributed Subgradient Optimization Algorithms

Privacy-Preserving Distributed Online Optimization Over Unbalanced Digraphs via Subgradient Rescaling

Differential Privacy in Distributed Optimization with Gradient Tracking

On the Privacy of Noisy Stochastic Gradient Descent for Convex Optimization

To Shuffle or not to Shuffle: Auditing DP-SGD with Shuffling

ReSQueing Parallel and Private Stochastic Convex Optimization

Distributed Least-Squares Optimization Solvers with Differential Privacy

Optimal Rates for $O(1)$-Smooth DP-SCO with a Single Epoch and Large Batches

Differentially private stochastic gradient descent with low-noise

Decentralized Nonconvex Optimization with Guaranteed Privacy and Accuracy

Empirical Risk Minimization with Shuffled SGD: A Primal-Dual Perspective and Improved Bounds

Private and Federated Stochastic Convex Optimization: Efficient Strategies for Centralized Systems