Abstract:In multi-agent reinforcement learning, optimal control with robustness guarantees are critical for its deployment in real world. However, existing methods face challenges related to sample complexity, training instability, potential suboptimal Nash Equilibrium convergence and non-robustness to multiple perturbations. In this paper, we propose a unified framework for learning \emph{stochastic} policies to resolve these issues. We embed cooperative MARL problems into probabilistic graphical models, from which we derive the maximum entropy (MaxEnt) objective optimal for MARL. Based on the MaxEnt framework, we propose \emph{Heterogeneous-Agent Soft Actor-Critic} (HASAC) algorithm. Theoretically, we prove the monotonic improvement and convergence to \emph{quantal response equilibrium} (QRE) properties of HASAC. Furthermore, HASAC is provably robust against a wide range of real-world uncertainties, including perturbations in rewards, environment dynamics, states, and actions. Finally, we generalize a unified template for MaxEnt algorithmic design named \emph{Maximum Entropy Heterogeneous-Agent Mirror Learning} (MEHAML), which provides any induced method with the same guarantees as HASAC. We evaluate HASAC on seven benchmarks: Bi-DexHands, Multi-Agent MuJoCo, Pursuit-Evade, StarCraft Multi-Agent Challenge, Google Research Football, Multi-Agent Particle Environment, Light Aircraft Game. Results show that HASAC consistently outperforms strong baselines in 34 out of 38 tasks, exhibiting improved training stability, better sample efficiency and sufficient exploration. The robustness of HASAC was further validated when encountering uncertainties in rewards, dynamics, states, and actions of 14 magnitudes, and real-world deployment in a multi-robot arena against these four types of uncertainties. See our page at \url{<a class="link-external link-https" href="https://sites.google.com/view/meharl" rel="external noopener nofollow">this https URL</a>}.

Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation

Sample-Efficient Multi-Agent RL: an Optimization Perspective.

Off-Beat Multi-Agent Reinforcement Learning

Learning and Calibrating Heterogeneous Bounded Rational Market Behaviour with Multi-Agent Reinforcement Learning

Multiagent Reinforcement Learning with Unshared Value Functions.

Hybrid Information-driven Multi-agent Reinforcement Learning

Accelerating Multiagent Reinforcement Learning by Equilibrium Transfer

Heterogeneous-Agent Reinforcement Learning

Robust Multi-Agent Control via Maximum Entropy Heterogeneous-Agent Reinforcement Learning

Provable Memory Efficient Self-Play Algorithm for Model-free Reinforcement Learning

Multiexperience-Assisted Efficient Multiagent Reinforcement Learning

Safe Multi-Agent Reinforcement Learning with Convergence to Generalized Nash Equilibrium

Efficient Multi-agent Reinforcement Learning by Planning

Multi-Agent Reinforcement Learning from Human Feedback: Data Coverage and Algorithmic Techniques

Multi-Agent Trust Region Learning

Decentralized Multi-Agent Reinforcement Learning: An Off-Policy Method

Combining Tree-Search, Generative Models, and Nash Bargaining Concepts in Game-Theoretic Reinforcement Learning

Model-based Reinforcement Learning for Decentralized Multiagent Rendezvous

Model-based Multi-agent Reinforcement Learning: Recent Progress and Prospects

Risk-Sensitive Multi-Agent Reinforcement Learning in Network Aggregative Markov Games

Policy Evaluation and Seeking for Multi-Agent Reinforcement Learning Via Best Response