Abstract:This paper considers a novel multi-agent linear stochastic approximation algorithm driven by Markovian noise and general consensus-type interaction, in which each agent evolves according to its local stochastic approximation process which depends on the information from its neighbors. The interconnection structure among the agents is described by a time-varying directed graph. While the convergence of consensus-based stochastic approximation algorithms when the interconnection among the agents is described by doubly stochastic matrices (at least in expectation) has been studied, less is known about the case when the interconnection matrix is simply stochastic. For any uniformly strongly connected graph sequences whose associated interaction matrices are stochastic, the paper derives finite-time bounds on the mean-square error, defined as the deviation of the output of the algorithm from the unique equilibrium point of the associated ordinary differential equation. For the case of interconnection matrices being stochastic, the equilibrium point can be any unspecified convex combination of the local equilibria of all the agents in the absence of communication. Both the cases with constant and time-varying step-sizes are considered. In the case when the convex combination is required to be a straight average and interaction between any pair of neighboring agents may be uni-directional, so that doubly stochastic matrices cannot be implemented in a distributed manner, the paper proposes a push-sum-type distributed stochastic approximation algorithm and provides its finite-time bound for the time-varying step-size case by leveraging the analysis for the consensus-type algorithm with stochastic matrices and developing novel properties of the push-sum algorithm. Distributed temporal difference learning is discussed as an illustrative application.

Convergence of a Hebbian-type learning algorithm

Convergence Analysis of General Neural Networks under Almost Periodic Stimuli

Convergence Analysis of General Evolutionary Algorithms

A Complex Hebbian-type Learning Algorithm with Application to Complex Adaptive ⅡR Filtering

Stochastic Averaging of Quasi Linear Systems Subject to Multi- Time-Delayed Feedback Control and Wide-Band Random Excitation

Stochastic averaging of quasi-integrable Hamiltonian systems under combined harmonic and white noise excitations

Iterative learning based convergence analysis for nonlinear impulsive differential inclusion systems with randomly varying trial lengths

An algorithm for frequency estimation of signals composed of multiple single-tones

Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noise

Convergence dynamics of stochastic Cohen-Grossberg neural networks with unbounded distributed delays

Global Exponential Convergence Analysis of Delayed Neural Networks with Time-Varying Delays

Convergent dynamics analysis of neural networks: A decomposition approach

Linear convergence of CQ algorithms and applications in gene regulatory network inference

Finite-Time Error Bounds for Distributed Linear Stochastic Approximation

Convergence of Langevin-simulated annealing algorithms with multiplicative noise

Convergence Theorems for the Kohonen Feature Mapping Algorithms with VLRPs

Global exponential convergence and synchronization for exponential numerical competitive neural networks with different time scales and fuzzy logic

Final Iteration Convergence Bound of Q-Learning: Switching System Approach

A General Framework for Analyzing Stochastic Dynamics in Learning Algorithms

Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features

Fast and Strong Convergence of Online Learning Algorithms.