Abstract:This paper considers a novel multi-agent linear stochastic approximation algorithm driven by Markovian noise and general consensus-type interaction, in which each agent evolves according to its local stochastic approximation process which depends on the information from its neighbors. The interconnection structure among the agents is described by a time-varying directed graph. While the convergence of consensus-based stochastic approximation algorithms when the interconnection among the agents is described by doubly stochastic matrices (at least in expectation) has been studied, less is known about the case when the interconnection matrix is simply stochastic. For any uniformly strongly connected graph sequences whose associated interaction matrices are stochastic, the paper derives finite-time bounds on the mean-square error, defined as the deviation of the output of the algorithm from the unique equilibrium point of the associated ordinary differential equation. For the case of interconnection matrices being stochastic, the equilibrium point can be any unspecified convex combination of the local equilibria of all the agents in the absence of communication. Both the cases with constant and time-varying step-sizes are considered. In the case when the convex combination is required to be a straight average and interaction between any pair of neighboring agents may be uni-directional, so that doubly stochastic matrices cannot be implemented in a distributed manner, the paper proposes a push-sum-type distributed stochastic approximation algorithm and provides its finite-time bound for the time-varying step-size case by leveraging the analysis for the consensus-type algorithm with stochastic matrices and developing novel properties of the push-sum algorithm. Distributed temporal difference learning is discussed as an illustrative application.

Convergence and Accuracy Analysis for A Distributed Static State Estimator Based on Gaussian Belief Propagation

Design and Analysis of Distributed State Estimation Algorithms Based on Belief Propagation and Applications in Smart Grids

Distributed Verification of Belief Precisions Convergence in Gaussian Belief Propagation

Distributed State Estimation for Heterogeneous Sensor Networks

Distributed Parameter Estimation with Random Observation Matrices and Communication Graphs

Convergence-Guaranteed Parametric Bayesian Distributed Cooperative Localization

Distributed State Estimation for Spatially Interconnected Systems and Its Convergence Analysis

Distributed Gauss-Newton Method for AC State Estimation: A Belief Propagation Approach

Accuracy analysis for distributed dynamic state estimation in large-scale systems with a cyclic network graph

Event-triggered Distributed Dynamic State Estimation with Imperfect Measurements over a Finite Horizon

Fixed Points of Gaussian Belief Propagation and Relation to Convergence

Convergence Analysis of Gaussian SPAWN under High-Order Graphical Models.

Distributed State Estimation for Uncertain Linear Systems with a Recursive Architecture

Understanding belief propagation and its generalizations

Finite-Time Error Bounds for Distributed Linear Stochastic Approximation

Gaussian belief propagation for distributed simultaneous localization and tracking in wireless sensor networks

Distributed Bayesian Estimation of Linear Models With Unknown Observation Covariances.

Distributed Estimation for Moving Target Based on State-Consensus Strategy.

Sequential estimation of linear models in distributed settings

Convergence Analysis of Gaussian Belief Propagation under High-Order Factorization and Asynchronous Scheduling

Linear Convergence of Distributed Estimation with Constraints and Communication Delays