Abstract:SIAM Journal on Optimization, Volume 34, Issue 1, Page 946-976, March 2024. We study a new two-time-scale stochastic gradient method for solving optimization problems, where the gradients are computed with the aid of an auxiliary variable under samples generated by time-varying Markov random processes controlled by the underlying optimization variable. These time-varying samples make gradient directions in our update biased and dependent, which can potentially lead to the divergence of the iterates. In our two-time-scale approach, one scale is to estimate the true gradient from these samples, which is then used to update the estimate of the optimal solution. While these two iterates are implemented simultaneously, the former is updated "faster" (using bigger step sizes) than the latter (using smaller step sizes). Our first contribution is to characterize the finite-time complexity of the proposed two-time-scale stochastic gradient method. In particular, we provide explicit formulas for the convergence rates of this method under different structural assumptions, namely, strong convexity, the Polyak–Łojasiewicz condition, and general nonconvexity. We apply our framework to policy optimization problems in control and reinforcement learning. First, we look at the infinite-horizon average-reward Markov decision process with finite state and action spaces and derive a convergence rate of [math] for the online actor-critic algorithm under function approximation, which recovers the best known rate derived specifically for this problem. Second, we study the linear-quadratic regulator and show that an online actor-critic method converges with rate [math]. Third, we use the actor-critic algorithm to solve the policy optimization problem in an entropy regularized Markov decision process, where we also establish a convergence of [math]. The results we derive for both the second and third problems are novel and previously unknown in the literature. Finally, we briefly present the application of our framework to gradient-based policy evaluation algorithms in reinforcement learning.

Stochastic Inertial Dynamics Via Time Scaling and Averaging

An SDE Perspective on Stochastic Inertial Gradient Dynamics with Time-Dependent Viscosity and Geometric Damping

Data-Driven Stochastic Averaging

Stochastic Averaging In Parametric Regions Near Separatrices Of Integrability

A Novel Method for Evaluating the Averaged Drift and Diffusion Coefficients of High DOF Quasi-Non-integrable Hamiltonian Systems

Stochastic averaging for quasi-integrable hamiltonian systems with variable mass

Inertial Methods with Viscous and Hessian driven Damping for Non-Convex Optimization

Stochastic Averaging for A Class of Two-Time-Scale Systems of Stochastic Partial Differential Equations

Fast convex optimization via closed-loop time scaling of gradient dynamics

Stochastic Optimization of Large-Scale Parametrized Dynamical Systems

A fast continuous time approach with time scaling for nonsmooth convex optimization

Second Order Dynamics Featuring Tikhonov Regularization and Time Scaling

Averaging principles for time-inhomogeneous multi-scale SDEs via nonautonomous Poisson equations

Stochastic Averaging in Continuous Time and Its Applications to Extremum Seeking

"Second-Order Primal" + "First-Order Dual" Dynamical Systems With Time Scaling for Linear Equality Constrained Convex Optimization Problems

The random timestep Euler method and its continuous dynamics

A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learning

Stochastic Averaging for the Non-Autonomous Mixed Stochastic Differential Equations with Locally Lipschitz Coefficients

On the study of slow-fast dynamics, when the fast process has multiple invariant measures

Convergence of Inertial Dynamics Driven by Sums of Potential and Nonpotential Operators with Implicit Newton-Like Damping

Fast convergence of dynamical ADMM via time scaling of damped inertial dynamics