Abstract:SIAM Journal on Optimization, Volume 34, Issue 1, Page 946-976, March 2024. We study a new two-time-scale stochastic gradient method for solving optimization problems, where the gradients are computed with the aid of an auxiliary variable under samples generated by time-varying Markov random processes controlled by the underlying optimization variable. These time-varying samples make gradient directions in our update biased and dependent, which can potentially lead to the divergence of the iterates. In our two-time-scale approach, one scale is to estimate the true gradient from these samples, which is then used to update the estimate of the optimal solution. While these two iterates are implemented simultaneously, the former is updated "faster" (using bigger step sizes) than the latter (using smaller step sizes). Our first contribution is to characterize the finite-time complexity of the proposed two-time-scale stochastic gradient method. In particular, we provide explicit formulas for the convergence rates of this method under different structural assumptions, namely, strong convexity, the Polyak–Łojasiewicz condition, and general nonconvexity. We apply our framework to policy optimization problems in control and reinforcement learning. First, we look at the infinite-horizon average-reward Markov decision process with finite state and action spaces and derive a convergence rate of [math] for the online actor-critic algorithm under function approximation, which recovers the best known rate derived specifically for this problem. Second, we study the linear-quadratic regulator and show that an online actor-critic method converges with rate [math]. Third, we use the actor-critic algorithm to solve the policy optimization problem in an entropy regularized Markov decision process, where we also establish a convergence of [math]. The results we derive for both the second and third problems are novel and previously unknown in the literature. Finally, we briefly present the application of our framework to gradient-based policy evaluation algorithms in reinforcement learning.

Convergence of Batch Asynchronous Stochastic Approximation With Applications to Reinforcement Learning

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

Asynchronous Stochastic Approximation and Average-Reward Reinforcement Learning

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications

Stochastic Approximation with Delayed Updates: Finite-Time Rates under Markovian Sampling

Stochastic Approximation with Unbounded Markovian Noise: A General-Purpose Theorem

Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learning

Tight Finite Time Bounds of Two-Time-Scale Linear Stochastic Approximation with Markovian Noise

A Lyapunov Theory for Finite-Sample Guarantees of Asynchronous Q-Learning and TD-Learning Variants

Finite-Time Convergence Rates of Decentralized Local Markovian Stochastic Approximation

The Curse of Memory in Stochastic Approximation: Extended Version

Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic Optimization

Asynchronous Stochastic Approximation Algorithms for Networked Systems: Regime-Switching Topologies and Multiscale Structure.

Convergence Rates of Accelerated Markov Gradient Descent with Applications in Reinforcement Learning

Joint Stochastic Approximation and Its Application to Learning Discrete Latent Variable Models.

Constant Stepsize Q-learning: Distributional Convergence, Bias and Extrapolation

DASA: Delay-Adaptive Multi-Agent Stochastic Approximation

Finite-Time Error Bounds of Biased Stochastic Approximation With Application to TD-Learning

A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learning

Convergence of Asynchronous Distributed Gradient Methods over Stochastic Networks

Markovian Foundations for Quasi-Stochastic Approximation in Two Timescales: Extended Version