Abstract:Multi-Agent Reinforcement Learning involves agents that learn together in a shared environment, leading to emergent dynamics sensitive to initial conditions and parameter variations. A Dynamical Systems approach, which studies the evolution of multi-component systems over time, has uncovered some of the underlying dynamics by constructing deterministic approximation models of stochastic algorithms. In this work, we demonstrate that even in the simplest case of independent Q-learning with a Boltzmann exploration policy, significant discrepancies arise between the actual algorithm and previous approximations. We elaborate why these models actually approximate interesting variants rather than the original incremental algorithm. To explain the discrepancies, we introduce a new discrete-time approximation model that explicitly accounts for agents' update frequencies within the learning process and show that its dynamics fundamentally differ from the simplified dynamics of prior models. We illustrate the usefulness of our approach by applying it to the question of spontaneous cooperation in social dilemmas, specifically the Prisoner's Dilemma as the simplest case study. We identify conditions under which the learning behaviour appears as long-term stable cooperation from an external perspective. However, our model shows that this behaviour is merely a metastable transient phase and not a true equilibrium, making it exploitable. We further exemplify how specific parameter settings can significantly exacerbate the moving target problem in independent learning. Through a systematic analysis of our model, we show that increasing the discount factor induces oscillations, preventing convergence to a joint policy. These oscillations arise from a supercritical Neimark-Sacker bifurcation, which transforms the unique stable fixed point into an unstable focus surrounded by a stable limit cycle.

Hypergraph-Based Model for Modelling Multi-agent Q-Learning Dynamics in Public Goods Games

A Formal Model for Multiagent Q-Learning Dynamics on Regular Graphs.

Learning and Propagation: Evolutionary Dynamics in Spatial Public Goods Games Through Combined Q-learning and Fermi Rule

A Pair-Approximation Method for Modelling the Dynamics of Multi-Agent Stochastic Games.

Cooperation in Public Goods Games: Leveraging Other-Regarding Reinforcement Learning on Hypergraphs

Hypergraphon mean field games

Dynamically Analyzing Cell Interactions in Biological Environments Using Multiagent Social Learning Framework

A Game-Theoretic Learning Model in Multi-Agent Systems

Modelling the Dynamics of Regret Minimization in Large Agent Populations: a Master Equation Approach.

Deterministic Model of Incremental Multi-Agent Boltzmann Q-Learning: Transient Cooperation, Metastability, and Oscillations

Network Characteristic Control of Social Dilemmas in a Public Good Game: Numerical Simulation of Agent-Based Nonlinear Dynamics

Asymptotic Convergence and Performance of Multi-Agent Q-Learning Dynamics

Improving Quantal Cognitive Hierarchy Model Through Iterative Population Learning

Collective Emergence in A Multi-Agent Network with Adaptive Games

Beyond Strict Competition: Approximate Convergence of Multi Agent Q-Learning Dynamics

Non-cooperative Multi-agent Systems with Exploring Agents

Accelerating Nash Q-Learning with Graphical Game Representation and Equilibrium Solving

Evolutionary Multi-agent Reinforcement Learning in Group Social Dilemmas

Stability of Multi-Agent Learning in Competitive Networks: Delaying the Onset of Chaos

Multi-agent reinforcement learning in the all-or-nothing public goods game on networks

Evolutionary Game Dynamics of Multiagent Systems on Multiple Community Networks.