Abstract:In multi-agent reinforcement learning, optimal control with robustness guarantees are critical for its deployment in real world. However, existing methods face challenges related to sample complexity, training instability, potential suboptimal Nash Equilibrium convergence and non-robustness to multiple perturbations. In this paper, we propose a unified framework for learning \emph{stochastic} policies to resolve these issues. We embed cooperative MARL problems into probabilistic graphical models, from which we derive the maximum entropy (MaxEnt) objective optimal for MARL. Based on the MaxEnt framework, we propose \emph{Heterogeneous-Agent Soft Actor-Critic} (HASAC) algorithm. Theoretically, we prove the monotonic improvement and convergence to \emph{quantal response equilibrium} (QRE) properties of HASAC. Furthermore, HASAC is provably robust against a wide range of real-world uncertainties, including perturbations in rewards, environment dynamics, states, and actions. Finally, we generalize a unified template for MaxEnt algorithmic design named \emph{Maximum Entropy Heterogeneous-Agent Mirror Learning} (MEHAML), which provides any induced method with the same guarantees as HASAC. We evaluate HASAC on seven benchmarks: Bi-DexHands, Multi-Agent MuJoCo, Pursuit-Evade, StarCraft Multi-Agent Challenge, Google Research Football, Multi-Agent Particle Environment, Light Aircraft Game. Results show that HASAC consistently outperforms strong baselines in 34 out of 38 tasks, exhibiting improved training stability, better sample efficiency and sufficient exploration. The robustness of HASAC was further validated when encountering uncertainties in rewards, dynamics, states, and actions of 14 magnitudes, and real-world deployment in a multi-robot arena against these four types of uncertainties. See our page at \url{<a class="link-external link-https" href="https://sites.google.com/view/meharl" rel="external noopener nofollow">this https URL</a>}.

Heterogeneous Multi-Robot Cooperation With Asynchronous Multi-Agent Reinforcement Learning

Hierarchical Consensus-Based Multi-Agent Reinforcement Learning for Multi-Robot Cooperation Tasks

Heterogeneous-Agent Reinforcement Learning

Adaptive Individual Q-Learning-A Multiagent Reinforcement Learning Method for Coordination Optimization

Heterogeneous Policy Networks for Composite Robot Team Communication and Coordination

Multi-robot Social-aware Cooperative Planning in Pedestrian Environments Using Multi-agent Reinforcement Learning

Heterogeneous Multi-Robot Reinforcement Learning

Multiagent Reinforcement Learning With Heterogeneous Graph Attention Network

NeuronsMAE: A Novel Multi-Agent Reinforcement Learning Environment for Cooperative and Competitive Multi-Robot Tasks

Multi-Agent Concentrative Coordination with Decentralized Task Representation

Heterogeneous Multi-Agent Reinforcement Learning for Zero-Shot Scalable Collaboration

Relative Distributed Formation and Obstacle Avoidance with Multi-agent Reinforcement Learning

Multi-agent Reinforcement Learning with Deep Networks for Diverse Q-Vectors

Robust Multi-Agent Reinforcement Learning with Social Empowerment for Coordination and Communication

Communication-Efficient Soft Actor-Critic Policy Collaboration via Regulated Segment Mixture

Robust Multi-Agent Control via Maximum Entropy Heterogeneous-Agent Reinforcement Learning

MQE: Unleashing the Power of Interaction with Multi-agent Quadruped Environment

GHQ: Grouped Hybrid Q Learning for Heterogeneous Cooperative Multi-agent Reinforcement Learning

GHQ: grouped hybrid Q-learning for cooperative heterogeneous multi-agent reinforcement learning

Multiagent Reinforcement Learning with Unshared Value Functions.

Reinforcement Learning for Synchronization of Heterogeneous Multiagent Systems by Improved Q-Functions