Abstract:We consider the stochastic gradient method with random reshuffling ($\mathsf{RR}$) for tackling smooth nonconvex optimization problems. $\mathsf{RR}$ finds broad applications in practice, notably in training neural networks. In this work, we first investigate the concentration property of $\mathsf{RR}$'s sampling procedure and establish a new high probability sample complexity guarantee for driving the gradient (without expectation) below $\varepsilon$, which effectively characterizes the efficiency of a single $\mathsf{RR}$ execution. Our derived complexity matches the best existing in-expectation one up to a logarithmic term while imposing no additional assumptions nor changing $\mathsf{RR}$'s updating rule. Furthermore, by leveraging our derived high probability descent property and bound on the stochastic error, we propose a simple and computable stopping criterion for $\mathsf{RR}$ (denoted as $\mathsf{RR}$-$\mathsf{sc}$). This criterion is guaranteed to be triggered after a finite number of iterations, and then $\mathsf{RR}$-$\mathsf{sc}$ returns an iterate with its gradient below $\varepsilon$ with high probability. Moreover, building on the proposed stopping criterion, we design a perturbed random reshuffling method ($\mathsf{p}$-$\mathsf{RR}$) that involves an additional randomized perturbation procedure near stationary points. We derive that $\mathsf{p}$-$\mathsf{RR}$ provably escapes strict saddle points and efficiently returns a second-order stationary point with high probability, without making any sub-Gaussian tail-type assumptions on the stochastic gradient errors. Finally, we conduct numerical experiments on neural network training to support our theoretical findings.

Distributed Random Reshuffling Over Networks

Distributed Random Reshuffling Methods with Improved Convergence

A Distributed Algorithm for Resource Allocation Over Dynamic Digraphs.

Distributed Robust Optimization in Networked System.

Distributed Stochastic Algorithm for Global Optimization in Networked System

A Push-Pull Gradient Method for Distributed Optimization in Networks.

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

Distributed Zeroth-Order Optimization: Convergence Rates That Match Centralized Counterpart

High Probability Guarantees for Random Reshuffling

A Distributed Gradient Algorithm Based on Randomized Block-Coordinate and Projection-Free over Networks

An Exponentially Convergent Distributed Algorithm for Resource Allocation Problem

Variance-reduced Reshuffling Gradient Descent for Nonconvex Optimization: Centralized and Distributed Algorithms

Random Gradient Extrapolation for Distributed and Stochastic Optimization

A Deep Reinforcement Learning Approach to Efficient Distributed Optimization

DISH: A Distributed Hybrid Optimization Method Leveraging System Heterogeneity

Distributed Stochastic Proximal Algorithm with Random Reshuffling for Non-Smooth Finite-Sum Optimization

Byzantine-Resilient Distributed Optimization of Multi-Dimensional Functions

Decentralized projected Riemannian stochastic recursive momentum method for smooth optimization on compact submanifolds

Accelerating Distributed Stochastic Optimization via Self-Repellent Random Walks

Accelerated Distributed Aggregative Optimization