Abstract:Large-scale multi-agent systems are often deployed across wide geographic areas, where agents interact with heterogeneous environments. There is an emerging interest in understanding the role of heterogeneity in the performance of the federated versions of classic reinforcement learning algorithms. In this paper, we study synchronous federated Q-learning, which aims to learn an optimal Q-function by having $K$ agents average their local Q-estimates per $E$ iterations. We observe an interesting phenomenon on the convergence speeds in terms of $K$ and $E$. Similar to the homogeneous environment settings, there is a linear speed-up concerning $K$ in reducing the errors that arise from sampling randomness. Yet, in sharp contrast to the homogeneous settings, $E>1$ leads to significant performance degradation. Specifically, we provide a fine-grained characterization of the error evolution in the presence of environmental heterogeneity, which decay to zero as the number of iterations $T$ increases. The slow convergence of having $E>1$ turns out to be fundamental rather than an artifact of our analysis. We prove that, for a wide range of stepsizes, the $\ell_{\infty}$ norm of the error cannot decay faster than $\Theta (E/T)$. In addition, our experiments demonstrate that the convergence exhibits an interesting two-phase phenomenon. For any given stepsize, there is a sharp phase-transition of the convergence: the error decays rapidly in the beginning yet later bounces up and stabilizes. Provided that the phase-transition time can be estimated, choosing different stepsizes for the two phases leads to faster overall convergence.

Asymptotic Convergence and Performance of Multi-Agent Q-Learning Dynamics

Beyond Strict Competition: Approximate Convergence of Multi Agent Q-Learning Dynamics

Stability of Multi-Agent Learning: Convergence in Network Games with Many Players

Stability of Multi-Agent Learning in Competitive Networks: Delaying the Onset of Chaos

Convergent Learning Algorithms for Unknown Reward Games

Multi-agent Learning in Cooperative General-Sum Games

Convergence of Heterogeneous Learning Dynamics in Zero-sum Stochastic Games

A novel multi-agent Q-learning algorithm in cooperative multi-agent system

Efficient off‐policy Q‐learning for multi‐agent systems by solving dual games

Independent and Decentralized Learning in Markov Potential Games

Multi-Agent Alternate Q-Learning.

Dynamically Analyzing Cell Interactions in Biological Environments Using Multiagent Social Learning Framework

Reinforcement Learning for Non-stationary Discrete-Time Linear–Quadratic Mean-Field Games in Multiple Populations

Asymptotic Extinction in Large Coordination Games

A Black-box Approach for Non-stationary Multi-agent Reinforcement Learning

Off-policy Q-learning: Solving Nash Equilibrium of Multi-Player Games with Network-Induced Delay and Unmeasured State.

Q-learning Solution for Optimal Consensus Control of Discrete-Time Multiagent Systems Using Reinforcement Learning

On the Convergence Rates of Federated Q-Learning across Heterogeneous Environments

Finite-Sample Analysis of Distributed Q-learning for Multi-Agent Networks.

Multiagent Soft Q-Learning

A Risk-Averse Equilibrium for Multi-Agent Systems