Abstract:This paper considers a novel multi-agent linear stochastic approximation algorithm driven by Markovian noise and general consensus-type interaction, in which each agent evolves according to its local stochastic approximation process which depends on the information from its neighbors. The interconnection structure among the agents is described by a time-varying directed graph. While the convergence of consensus-based stochastic approximation algorithms when the interconnection among the agents is described by doubly stochastic matrices (at least in expectation) has been studied, less is known about the case when the interconnection matrix is simply stochastic. For any uniformly strongly connected graph sequences whose associated interaction matrices are stochastic, the paper derives finite-time bounds on the mean-square error, defined as the deviation of the output of the algorithm from the unique equilibrium point of the associated ordinary differential equation. For the case of interconnection matrices being stochastic, the equilibrium point can be any unspecified convex combination of the local equilibria of all the agents in the absence of communication. Both the cases with constant and time-varying step-sizes are considered. In the case when the convex combination is required to be a straight average and interaction between any pair of neighboring agents may be uni-directional, so that doubly stochastic matrices cannot be implemented in a distributed manner, the paper proposes a push-sum-type distributed stochastic approximation algorithm and provides its finite-time bound for the time-varying step-size case by leveraging the analysis for the consensus-type algorithm with stochastic matrices and developing novel properties of the push-sum algorithm. Distributed temporal difference learning is discussed as an illustrative application.

Stochastic approximation for consensus over general digraphs with Markovian switches

Average consensus analysis of distributed inference with uncertain markovian transition probability

Average Consensus of Sensor Networks with Uncertain Markov Topologies

Consensus under Bounded Noise in Discrete Network Systems: an Algorithm with Fast Convergence and High Accuracy.

Consensus in Networks of Multiagents with Stochastically Switching Topologies and Time-Varying Delays

Finite-Time Error Bounds for Distributed Linear Stochastic Approximation

Dissipativity-Based Consensus Tracking of Singular Multiagent Systems With Switching Topologies and Communication Delays

Stochastic Dynamics of Noisy Average Consensus: Analysis and Optimization

Consensus in Networks of Multiagents with Switching Topologies Modeled As Adapted Stochastic Processes

Consensus for general multi-agent systems over random graphs.

Reaching ℒp Consensus in a Network of Multiagents with Stochastically Switching Topologies

Scaled consensus of multi-agent systems with switching topologies and communication noises

Distributed Finite-Time Consensus of Nonlinear Systems under Switching Topologies

Distributed consensus tracking of nonlinear MASs under stochastic switching topologies via aperiodically intermittent control

Adaptive mean‐square consensus of heterogeneous multiagent systems with stochastic switching topologies

Finite-time average consensus of directed second-order multi-agent systems with Markovian switching topology and impulsive disturbance

Consensus in networks of multiagents with cooperation and competition via stochastically switching topologies.

Cluster consensus of nonlinear multi-agent systems with Markovian switching topologies and communication noises

Stochastic Consensus Control Integrated With Performance Improvement: A Consensus Region-Based Approach

Products of Generalized Stochastic Matrices With Applications to Consensus Analysis in Networks of Multiagents With Delays

Sampled-Data Consensus over Random Networks