Abstract:Stochastic approximation (SA) and stochastic gradient descent (SGD) algorithms are work-horses for modern machine learning algorithms. Their constant stepsize variants are preferred in practice due to fast convergence behavior. However, constant step stochastic iterative algorithms do not converge asymptotically to the optimal solution, but instead have a stationary distribution, which in general cannot be analytically characterized. In this work, we study the asymptotic behavior of the appropriately scaled stationary distribution, in the limit when the constant stepsize goes to zero. Specifically, we consider the following three settings: (1) SGD algorithms with smooth and strongly convex objective, (2) linear SA algorithms involving a Hurwitz matrix, and (3) nonlinear SA algorithms involving a contractive operator. When the iterate is scaled by $1/\sqrt{\alpha}$, where $\alpha$ is the constant stepsize, we show that the limiting scaled stationary distribution is a solution of an integral equation. Under a uniqueness assumption (which can be removed in certain settings) on this equation, we further characterize the limiting distribution as a Gaussian distribution whose covariance matrix is the unique solution of a suitable Lyapunov equation. For SA algorithms beyond these cases, our numerical experiments suggest that unlike central limit theorem type results: (1) the scaling factor need not be $1/\sqrt{\alpha}$, and (2) the limiting distribution need not be Gaussian. Based on the numerical study, we come up with a formula to determine the right scaling factor, and make insightful connection to the Euler-Maruyama discretization scheme for approximating stochastic differential equations.

A Constant Step Stochastic Douglas-Rachford Algorithm with Application to Non Separable Regularizations

Algorithm 1 : Non-stationary Forward – Douglas – Rachford splitting

The degenerate variable metric proximal point algorithm and adaptive stepsizes for primal–dual Douglas–Rachford

A stochastic preconditioned Douglas-Rachford splitting method for saddle-point problems

Local Convergence Properties of Douglas–Rachford and Alternating Direction Method of Multipliers

A constant step Forward-Backward algorithm involving random maximal monotone operators

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

A parameterized Douglas-Rachford Splitting algorithm for nonconvex optimization

Stationary Behavior of Constant Stepsize SGD Type Algorithms: An Asymptotic Characterization

On the finite convergence of the Douglas-Rachford algorithm for solving (not necessarily convex) feasibility problems in Euclidean spaces

On the local convergence of the Douglas-Rachford algorithm

On the Douglas-Rachford and Peaceman-Rachford algorithms in the presence of uniform monotonicity and the absence of minimizers

Activity Identification and Local Linear Convergence of Douglas–Rachford/ADMM under Partial Smoothness

Randomized Douglas–Rachford Methods for Linear Systems: Improved Accuracy and Efficiency

Convergence Rates of Forward–Douglas–Rachford Splitting Method

New Douglas-Rachford algorithmic structures and their convergence analyses

An Inertial Parametric Douglas–Rachford Splitting Method for Nonconvex Problems

Convergence of Markov Chains for Constant Step-size Stochastic Gradient Descent with Separable Functions

An alternating iterative minimisation algorithm for the double-regularised total least square functional

Constant Step Stochastic Approximations Involving Differential Inclusions: Stability, Long-Run Convergence and Applications

Stochastic Primal–Dual Hybrid Gradient Algorithm with Adaptive Step Sizes