Abstract:Stochastic approximation (SA) with multiple coupled sequences has found broad applications in machine learning such as bilevel learning and reinforcement learning (RL). In this paper, we study the finite-time convergence of nonlinear SA with multiple coupled sequences. Different from existing multi-timescale analysis, we seek for scenarios where a fine-grained analysis can provide the tight performance guarantee for multi-sequence single-timescale SA (STSA). At the heart of our analysis is the smoothness property of the fixed points in multi-sequence SA that holds in many applications. When all sequences have strongly monotone increments, we establish the iteration complexity of $\mathcal{O}(\epsilon^{-1})$ to achieve $\epsilon$-accuracy, which improves the existing $\mathcal{O}(\epsilon^{-1.5})$ complexity for two coupled sequences. When all but the main sequence have strongly monotone increments, we establish the iteration complexity of $\mathcal{O}(\epsilon^{-2})$. The merit of our results lies in that applying them to stochastic bilevel and compositional optimization problems, as well as RL problems leads to either relaxed assumptions or improvements over their existing performance guarantees.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is related to the finite - time convergence analysis of Stochastic Approximation (SA) in the case of multi - sequence coupling. Specifically, the paper focuses on how to conduct a fine - grained analysis of nonlinear Stochastic Approximation under a single - time scale to provide more compact performance guarantees. The main problems of the paper can be summarized as the following two points: 1. **Can a faster convergence rate be proven in the nonlinear Stochastic Approximation of two coupled sequences?** - The existing dual - sequence SA analysis mainly focuses on the Two - Time - Scale (TTS) method. This method assumes that one sequence is updated on a faster time scale and the other sequence is updated on a slower time scale. However, the iterative complexity of this analysis method for nonlinear SA is not as good as that for linear SA. The paper proposes a single - time - scale method and proves that under certain conditions, nonlinear SA can achieve the same $O(\epsilon^{- 1})$ iterative complexity as linear SA. 2. **Can a convergence rate of multi - sequence SA independent of the number of sequences $N$ be established?** - In multi - sequence SA, as the number of sequences $N$ increases, the existing analysis methods usually lead to a worse convergence rate. By introducing the smoothness assumption of fixed points, the paper proves that the convergence rate of multi - sequence SA can be independent of $N$, thus solving this problem. ### Main contributions of the paper 1. **Single - time - scale analysis**: - Different from the existing dual - time - scale analysis, the paper establishes a unified single - time - scale analysis framework (STSA) for multi - sequence - coupled SA. When all sequences are strongly monotonically increasing, the paper improves the $O(\epsilon^{-1.5})$ iterative complexity of multi - sequence TTS - SA to $O(\epsilon^{-1})$; when all sequences except the main sequence are strongly monotonically increasing, it provides an $O(\epsilon^{-2})$ iterative complexity. 2. **Application to Stochastic Bi - level Optimization (SBO)**: - When applying the general results to the SBO problem, for strongly concave objective functions, the paper improves the best - known sample complexity from $\tilde{O}(\epsilon^{-1.5})$ to $\tilde{O}(\epsilon^{-1})$. For non - concave objective functions, the paper achieves the same sample complexity $O(\epsilon^{-2})$ as ALSET while relaxing the assumption that the upper - level gradient is bounded. 3. **Application to Stochastic Combinatorial Optimization (SCO)**: - For multi - level SCO problems, the paper improves the sample complexity of the $\alpha$-TSCGD method based on multi - sequence SA from $O(\epsilon^{-(N + 5)/4})$ and $O(\epsilon^{-(N + 8)/4})$ to level - independent $\tilde{O}(\epsilon^{-1})$ and $O(\epsilon^{-2})$. 4. **Application to policy optimization in reinforcement learning**: - When applied to the actor - critic method, the paper achieves the same $O(\epsilon^{-2})$ sample complexity as ALSET - AC while relaxing the unverifiable assumption of the stationary distribution of the Markov chain; when applied to the meta - policy gradient, it improves the $O(\epsilon^{-4})$ sample complexity of SG - MRL to $O(\epsilon^{-2})$. ### Conclusion By introducing the smoothness assumption of fixed points, the paper successfully solves the single - time - scale analysis problem of multi - sequence - coupled SA and proves that its convergence rate can be independent of the number of sequences $N$. These results not only improve the performance guarantees of existing methods but also expand the application range of multi - sequence SA, especially in fields such as stochastic bi - level optimization, stochastic combinatorial optimization, and reinforcement learning.

A Single-Timescale Analysis For Stochastic Approximation With Multiple Coupled Sequences

Single-Timescale Multi-Sequence Stochastic Approximation Without Fixed Point Smoothness: Theories and Applications

Finite-Time Decoupled Convergence in Nonlinear Two-Time-Scale Stochastic Approximation

Tight Finite Time Bounds of Two-Time-Scale Linear Stochastic Approximation with Markovian Noise

Two-Timescale Linear Stochastic Approximation: Constant Stepsizes Go a Long Way

Markovian Foundations for Quasi-Stochastic Approximation in Two Timescales: Extended Version

Online Successive Convex Approximation for Two-Stage Stochastic Nonconvex Optimization

The Collusion of Memory and Nonlinearity in Stochastic Approximation With Constant Stepsize

Accelerated Multi-Time-Scale Stochastic Approximation: Optimal Complexity and Applications in Reinforcement Learning and Multi-Agent Games

Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic Optimization

A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learning

Prelimit Coupling and Steady-State Convergence of Constant-stepsize Nonsmooth Contractive SA

Central Limit Theorem for Two-Timescale Stochastic Approximation with Markovian Noise: Theory and Applications

Stochastic Approximation of Smooth and Strongly Convex Functions: Beyond the O(1/T) Convergence Rate.

Stochastic Approximation with Delayed Updates: Finite-Time Rates under Markovian Sampling

Fast Two-Time-Scale Stochastic Gradient Method with Applications in Reinforcement Learning

Finite-Time Convergence Rates of Decentralized Local Markovian Stochastic Approximation

Stochastic Approximation with Unbounded Markovian Noise: A General-Purpose Theorem

The Curse of Memory in Stochastic Approximation: Extended Version

Stochastic Approximation of Smooth and Strongly Convex Functions: Beyond the Convergence Rate