Abstract:We study a new two-time-scale stochastic gradient method for solving optimization problems, where the gradients are computed with the aid of an auxiliary variable under samples generated by time-varying MDPs controlled by the underlying optimization variable. These time-varying samples make gradient directions in our update biased and dependent, which can potentially lead to the divergence of the iterates. In our two-time-scale approach, one scale is to estimate the true gradient from these samples, which is then used to update the estimate of the optimal solution. While these two iterates are implemented simultaneously, the former is updated "faster" than the latter. Our first contribution is to characterize the finite-time complexity of the proposed two-time-scale stochastic gradient method. In particular, we provide explicit formulas for the convergence rates of this method under different structural assumptions, namely, strong convexity, PL condition, and general non-convexity. We apply our framework to various policy optimization problems. First, we look at the infinite-horizon average-reward MDP with finite state and action spaces and derive a convergence rate of $O(k^{-2/5})$ for the online actor-critic algorithm under function approximation, which recovers the best known rate derived specifically for this problem. Second, we study the linear-quadratic regulator and show that an online actor-critic method converges with rate $O(k^{-2/3})$. Third, we use the actor-critic algorithm to solve the policy optimization problem in an entropy regularized Markov decision process, where we also establish a convergence of $O(k^{-2/3})$. The results we derive for both the second and third problem are novel and previously unknown in the literature. Finally, we briefly present the application of our framework to gradient-based policy evaluation algorithms in reinforcement learning.

Actor-Critic or Critic-Actor? A Tale of Two Time Scales

Two-Timescale Critic-Actor for Average Reward MDPs with Function Approximation

On the sample complexity of actor-critic method for reinforcement learning with function approximation

Finite-Sample Analysis of Off-Policy Natural Actor–Critic With Linear Function Approximation

Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximation

Compatible Gradient Approximations for Actor-Critic Algorithms

Finite-Time Analysis of Three-Timescale Constrained Actor-Critic and Constrained Natural Actor-Critic Algorithms

An Approximate Policy Iteration Viewpoint of Actor-Critic Algorithms

On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Cost

A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learning

Solving Time-Continuous Stochastic Optimal Control Problems: Algorithm Design and Convergence Analysis of Actor-Critic Flow

Improved Sample Complexity for Global Convergence of Actor-Critic Algorithms

Convergence Rates of Online Critic Value Function Approximation in Native Spaces

Addressing Function Approximation Error in Actor-Critic Methods

On the Global Convergence of Natural Actor-Critic with Two-layer Neural Network Parametrization

Value Improved Actor Critic Algorithms

Single Time-scale Actor-critic Method to Solve the Linear Quadratic Regulator with Convergence Guarantees

Boosting the Actor with Dual Critic

Warm-Start Actor-Critic: From Approximation Error to Sub-optimality Gap

Finite-Time Complexity of Online Primal-Dual Natural Actor-Critic Algorithm for Constrained Markov Decision Processes