Abstract:This paper considers a novel multi-agent linear stochastic approximation algorithm driven by Markovian noise and general consensus-type interaction, in which each agent evolves according to its local stochastic approximation process which depends on the information from its neighbors. The interconnection structure among the agents is described by a time-varying directed graph. While the convergence of consensus-based stochastic approximation algorithms when the interconnection among the agents is described by doubly stochastic matrices (at least in expectation) has been studied, less is known about the case when the interconnection matrix is simply stochastic. For any uniformly strongly connected graph sequences whose associated interaction matrices are stochastic, the paper derives finite-time bounds on the mean-square error, defined as the deviation of the output of the algorithm from the unique equilibrium point of the associated ordinary differential equation. For the case of interconnection matrices being stochastic, the equilibrium point can be any unspecified convex combination of the local equilibria of all the agents in the absence of communication. Both the cases with constant and time-varying step-sizes are considered. In the case when the convex combination is required to be a straight average and interaction between any pair of neighboring agents may be uni-directional, so that doubly stochastic matrices cannot be implemented in a distributed manner, the paper proposes a push-sum-type distributed stochastic approximation algorithm and provides its finite-time bound for the time-varying step-size case by leveraging the analysis for the consensus-type algorithm with stochastic matrices and developing novel properties of the push-sum algorithm. Distributed temporal difference learning is discussed as an illustrative application.

Distributed Stochastic Approximation: Weak Convergence and Network Design

Distributed Stochastic Optimization for Weakly Coupled Systems with Applications to Wireless Communications

Performance of a Distributed Stochastic Approximation Algorithm

Distributed Stochastic Algorithm for Global Optimization in Networked System

Distributed Stochastic Optimization With Unbounded Subgradients Over Randomly Time-Varying Networks

Distributed Constrained Stochastic Subgradient Algorithms Based on Random Projection and Asynchronous Broadcast over Networks

Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms

Distributed estimation over complex networks

Distributed Linear Equations over Random Networks

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

Convergence of Asynchronous Distributed Gradient Methods over Stochastic Networks

Distributed ADMM for Time-Varying Communication Networks

Finite-Time Error Bounds for Distributed Linear Stochastic Approximation

Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis

Distributed fixed-point algorithms for dynamic convex optimization over decentralized and unbalanced wireless networks

Discretized Distributed Optimization over Dynamic Digraphs

Asymptotic Properties of Primal-Dual Algorithm for Distributed Stochastic Optimization over Random Networks with Imperfect Communications.

ADMM-Tracking Gradient for Distributed Optimization over Asynchronous and Unreliable Networks

Asymptotic Properties of Primal-Dual Algorithm for Distributed Stochastic Optimization Over Random Networks

Accelerated Distributed Stochastic Non-Convex Optimization over Time-Varying Directed Networks

Distributed Stochastic Subgradient Projection Algorithms for Convex Optimization