Abstract:Multi-time-scale stochastic approximation is an iterative algorithm for finding the fixed point of a set of $N$ coupled operators given their noisy samples. It has been observed that due to the coupling between the decision variables and noisy samples of the operators, the performance of this method decays as $N$ increases. In this work, we develop a new accelerated variant of multi-time-scale stochastic approximation, which significantly improves the convergence rates of its standard counterpart. Our key idea is to introduce auxiliary variables to dynamically estimate the operators from their samples, which are then used to update the decision variables. These auxiliary variables help not only to control the variance of the operator estimates but also to decouple the sampling noise and the decision variables. This allows us to select more aggressive step sizes to achieve an optimal convergence rate. Specifically, under a strong monotonicity condition, we show that for any value of $N$ the $t^{\text{th}}$ iterate of the proposed algorithm converges to the desired solution at a rate $\widetilde{O}(1/t)$ when the operator samples are generated from a single from Markov process trajectory. A second contribution of this work is to demonstrate that the objective of a range of problems in reinforcement learning and multi-agent games can be expressed as a system of fixed-point equations. As such, the proposed approach can be used to design new learning algorithms for solving these problems. We illustrate this observation with numerical simulations in a multi-agent game and show the advantage of the proposed method over the standard multi-time-scale stochastic approximation algorithm.

On the Convergence of Single-Timescale Multi-Sequence Stochastic Approximation Without Fixed Point Smoothness

Single-Timescale Multi-Sequence Stochastic Approximation Without Fixed Point Smoothness: Theories and Applications

A Single-Timescale Analysis For Stochastic Approximation With Multiple Coupled Sequences

Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic Optimization

Finite-Time Decoupled Convergence in Nonlinear Two-Time-Scale Stochastic Approximation

Markovian Foundations for Quasi-Stochastic Approximation in Two Timescales: Extended Version

Two-Timescale Linear Stochastic Approximation: Constant Stepsizes Go a Long Way

Tight Finite Time Bounds of Two-Time-Scale Linear Stochastic Approximation with Markovian Noise

Stochastic Approximation of Smooth and Strongly Convex Functions: Beyond the O(1/T) Convergence Rate.

Decoupled Functional Central Limit Theorems for Two-Time-Scale Stochastic Approximation

Accelerated Multi-Time-Scale Stochastic Approximation: Optimal Complexity and Applications in Reinforcement Learning and Multi-Agent Games

Stochastic Approximation of Smooth and Strongly Convex Functions: Beyond the Convergence Rate

The Collusion of Memory and Nonlinearity in Stochastic Approximation With Constant Stepsize

The Modified MSA, a Gradient Flow and Convergence

Federated Multi-Sequence Stochastic Approximation with Local Hypergradient Estimation

Finite-Time Convergence Rates of Decentralized Local Markovian Stochastic Approximation

Dynamic stochastic approximation for multi-stage stochastic optimization

Convergence Analysis of a Smoothing SAA Method for a Stochastic Mathematical Program with Second-Order Cone Complementarity Constraints

Scalable Stochastic Alternating Direction Method of Multipliers.

Prelimit Coupling and Steady-State Convergence of Constant-stepsize Nonsmooth Contractive SA

Review of Two-timescale Stochastic Approximation Algorithms