Abstract:Motivated by applications in large-scale and multi-agent reinforcement learning, we study the non-asymptotic performance of stochastic approximation (SA) schemes with delayed updates under Markovian sampling. While the effect of delays has been extensively studied for optimization, the manner in which they interact with the underlying Markov process to shape the finite-time performance of SA remains poorly understood. In this context, our first main contribution is to show that under time-varying bounded delays, the delayed SA update rule guarantees exponentially fast convergence of the \emph{last iterate} to a ball around the SA operator's fixed point. Notably, our bound is \emph{tight} in its dependence on both the maximum delay $\tau_{max}$, and the mixing time $\tau_{mix}$. To achieve this tight bound, we develop a novel inductive proof technique that, unlike various existing delayed-optimization analyses, relies on establishing uniform boundedness of the iterates. As such, our proof may be of independent interest. Next, to mitigate the impact of the maximum delay on the convergence rate, we provide the first finite-time analysis of a delay-adaptive SA scheme under Markovian sampling. In particular, we show that the exponent of convergence of this scheme gets scaled down by $\tau_{avg}$, as opposed to $\tau_{max}$ for the vanilla delayed SA rule; here, $\tau_{avg}$ denotes the average delay across all iterations. Moreover, the adaptive scheme requires no prior knowledge of the delay sequence for step-size tuning. Our theoretical findings shed light on the finite-time effects of delays for a broad class of algorithms, including TD learning, Q-learning, and stochastic gradient descent under Markovian sampling.

Stochastic Analysis of LMS Algorithm with Delayed Block Coefficient Adaptation

Steady-state performance analysis for adaptive filters with error nonlinearities

A Unified Analysis Approach for LMS-based Variable Step-Size Algorithms

A Novel Systolic Structure Of Adaptive Filter Based On Parallel Delay Lms Algorithm

Speeding up VSLMS adaptation algorithms using dynamic adaptation gain: Analysis and Applications

Stochastic Approximation with Delayed Updates: Finite-Time Rates under Markovian Sampling

Stochastic Analysis of the Filtered-x LMS Algorithm for Active Noise Control

A New Noise-Assistant LMS Algorithm for Preventing the Stalling Effect

Stable Total Least Mean Square Adaptive Filter Algorithm

Parallel Nlms Filters With Stochastic Active Taps And Step-Sizes For Sparse System Identification

Research on a New LMS Algorithm with Variable Steplength

Convergence Analysis of the Modified Frequency-Domain Block LMS Algorithm with Guaranteed Optimal Steady State Performance

Dynamic variable step size LMS adaptation algorithms -- Application to adaptive feedforward noise attenuation

Anti-Interference Performance Improvement for Sigmoid Function Variable Step-Size LMS Adaptive Algorithm

Stability analysis of the bias compensated LMS algorithm

Parallel Delayed LMS Algorithm

A Novel LMS Adaptive Filtering Algorithm with Variable Step Size

New Variable Block-Size Frequency-Domain LMS Adaptive Algorithm

A New SPNLMS Algorithm Based on l∞-LMS

Combination of LMS Adaptive Filters with Coefficients Feedback

Simulation and Engineering Realization of Adaptive Filter Algorithm