Abstract:This paper considers a novel multi-agent linear stochastic approximation algorithm driven by Markovian noise and general consensus-type interaction, in which each agent evolves according to its local stochastic approximation process which depends on the information from its neighbors. The interconnection structure among the agents is described by a time-varying directed graph. While the convergence of consensus-based stochastic approximation algorithms when the interconnection among the agents is described by doubly stochastic matrices (at least in expectation) has been studied, less is known about the case when the interconnection matrix is simply stochastic. For any uniformly strongly connected graph sequences whose associated interaction matrices are stochastic, the paper derives finite-time bounds on the mean-square error, defined as the deviation of the output of the algorithm from the unique equilibrium point of the associated ordinary differential equation. For the case of interconnection matrices being stochastic, the equilibrium point can be any unspecified convex combination of the local equilibria of all the agents in the absence of communication. Both the cases with constant and time-varying step-sizes are considered. In the case when the convex combination is required to be a straight average and interaction between any pair of neighboring agents may be uni-directional, so that doubly stochastic matrices cannot be implemented in a distributed manner, the paper proposes a push-sum-type distributed stochastic approximation algorithm and provides its finite-time bound for the time-varying step-size case by leveraging the analysis for the consensus-type algorithm with stochastic matrices and developing novel properties of the push-sum algorithm. Distributed temporal difference learning is discussed as an illustrative application.

Random Asynchronous Iterations in Distributed Coordination Algorithms

Distributed Consensus of Multi-Agent Systems with Nonlinear Dynamics Via Adaptive Intermittent Control

Asynchronous Implementation of Distributed Coordination Algorithms: Conditions Using Partially Scrambling and Essentially Cyclic Matrices.

Delay-induced Discrete-Time Consensus

Asynchronous Stochastic Approximation Algorithms for Networked Systems: Regime-Switching Topologies and Multiscale Structure.

Study on Distributed Dynamic Average Consensus Algorithm

Finite-Time Error Bounds for Distributed Linear Stochastic Approximation

Distributed Optimal Consensus of Multi-Agent Systems: A Randomized Parallel Approach

Asynchronous Distributed Averaging Using Double Linear Iterations

Optimal Amortized Distributed Consensus

Consensus in Networks of Multiagents with Stochastically Switching Topologies and Time-Varying Delays

Asynchronous Consensus in Continuous-Time Multi-Agent Systems with Switching Topology and Time-Varying Delays

Distributed Control of Nonlinear Multiagent Systems With Asymptotic Consensus

Asynchronous Distributed Optimization with Delay-free Parameters

Sufficient Conditions on Distributed Averaging with Compound Noises and Fixed Topologies

Fully Distributed Event-Driven Adaptive Consensus of Unknown Linear Systems

On the Asynchronous Bipartite Consensus for Discrete-Time Second-Order Multi-Agent Systems with Switching Topologies

Consensus Problems Over Cooperation-Competition Random Switching Networks With Noisy Channels

Distributed Synchronization of Coupled Neural Networks Via Randomly Occurring Control.

Distributed Consensus of Stochastic Delayed Multi-agent Systems under Asynchronous Switching

Adaptive Algorithms for Synchronization, Consensus of Multi-Agents and Anti-Synchronization of Direct Complex Networks