Abstract:This paper considers a novel multi-agent linear stochastic approximation algorithm driven by Markovian noise and general consensus-type interaction, in which each agent evolves according to its local stochastic approximation process which depends on the information from its neighbors. The interconnection structure among the agents is described by a time-varying directed graph. While the convergence of consensus-based stochastic approximation algorithms when the interconnection among the agents is described by doubly stochastic matrices (at least in expectation) has been studied, less is known about the case when the interconnection matrix is simply stochastic. For any uniformly strongly connected graph sequences whose associated interaction matrices are stochastic, the paper derives finite-time bounds on the mean-square error, defined as the deviation of the output of the algorithm from the unique equilibrium point of the associated ordinary differential equation. For the case of interconnection matrices being stochastic, the equilibrium point can be any unspecified convex combination of the local equilibria of all the agents in the absence of communication. Both the cases with constant and time-varying step-sizes are considered. In the case when the convex combination is required to be a straight average and interaction between any pair of neighboring agents may be uni-directional, so that doubly stochastic matrices cannot be implemented in a distributed manner, the paper proposes a push-sum-type distributed stochastic approximation algorithm and provides its finite-time bound for the time-varying step-size case by leveraging the analysis for the consensus-type algorithm with stochastic matrices and developing novel properties of the push-sum algorithm. Distributed temporal difference learning is discussed as an illustrative application.

Distributed TD(0) with Almost No Communication

One-Shot Averaging for Distributed TD($λ$) Under Markov Sampling

Finite time analysis of temporal difference learning with linear function approximation: Tail averaging and regularisation

A primal-dual perspective for distributed TD-learning

Finite-Sample Analysis of Decentralized Temporal-Difference Learning with Linear Function Approximation

Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features

Distributed Dynamic Averaging Tracking Without Rate Measurements

Temporal Difference Learning as Gradient Splitting

Accelerated Gradient Temporal Difference Learning

Improved High-Probability Bounds for the Temporal Difference Learning Algorithm via Exponential Stability

A Simple Finite-Time Analysis of TD Learning with Linear Function Approximation

Baird Counterexample is Solved: with an example of How to Debug a Two-time-scale Algorithm

Finite-Time Analysis of Asynchronous Multi-Agent TD Learning

Statistical Inference for Temporal Difference Learning with Linear Function Approximation

A Concentration Bound for TD(0) with Function Approximation

Temporal Difference Learning with Compressed Updates: Error-Feedback meets Reinforcement Learning

Finite-Time Error Bounds for Distributed Linear Stochastic Approximation

Almost Sure Convergence of Average Reward Temporal Difference Learning

Fast Multi-Agent Temporal-Difference Learning via Homotopy Stochastic Primal-Dual Optimization

Geometric Insights into the Convergence of Nonlinear TD Learning

Statistical Efficiency of Distributional Temporal Difference Learning