Abstract:In multi-agent reinforcement learning, optimal control with robustness guarantees are critical for its deployment in real world. However, existing methods face challenges related to sample complexity, training instability, potential suboptimal Nash Equilibrium convergence and non-robustness to multiple perturbations. In this paper, we propose a unified framework for learning \emph{stochastic} policies to resolve these issues. We embed cooperative MARL problems into probabilistic graphical models, from which we derive the maximum entropy (MaxEnt) objective optimal for MARL. Based on the MaxEnt framework, we propose \emph{Heterogeneous-Agent Soft Actor-Critic} (HASAC) algorithm. Theoretically, we prove the monotonic improvement and convergence to \emph{quantal response equilibrium} (QRE) properties of HASAC. Furthermore, HASAC is provably robust against a wide range of real-world uncertainties, including perturbations in rewards, environment dynamics, states, and actions. Finally, we generalize a unified template for MaxEnt algorithmic design named \emph{Maximum Entropy Heterogeneous-Agent Mirror Learning} (MEHAML), which provides any induced method with the same guarantees as HASAC. We evaluate HASAC on seven benchmarks: Bi-DexHands, Multi-Agent MuJoCo, Pursuit-Evade, StarCraft Multi-Agent Challenge, Google Research Football, Multi-Agent Particle Environment, Light Aircraft Game. Results show that HASAC consistently outperforms strong baselines in 34 out of 38 tasks, exhibiting improved training stability, better sample efficiency and sufficient exploration. The robustness of HASAC was further validated when encountering uncertainties in rewards, dynamics, states, and actions of 14 magnitudes, and real-world deployment in a multi-robot arena against these four types of uncertainties. See our page at \url{<a class="link-external link-https" href="https://sites.google.com/view/meharl" rel="external noopener nofollow">this https URL</a>}.

Quantum Computing and Neuromorphic Computing for Safe, Reliable, and explainable Multi-Agent Reinforcement Learning: Optimal Control in Autonomous Robotics

Quantum Multi-Agent Reinforcement Learning for Autonomous Mobility Cooperation

Learning Observation-Based Certifiable Safe Policy for Decentralized Multi-Robot Navigation

Robust Optimization for Quantum Reinforcement Learning Control Using Partial Observations

Quantum Multi-Agent Reinforcement Learning for Aerial Ad-hoc Networks

Safe Multiagent Learning with Soft Constrained Policy Optimization in Real Robot Control

Brain-Inspired Agents for Quantum Reinforcement Learning

Safe Multi-Agent Reinforcement Learning for Multi-Robot Control

Software Simulation and Visualization of Quantum Multi-Drone Reinforcement Learning

Decentralized Safe and Scalable Multi-Agent Control under Limited Actuation

A Neuro-Symbolic Approach to Multi-Agent RL for Interpretability and Probabilistic Decision Making

Universal Quantum Control through Deep Reinforcement Learning

Optimization of Reinforcement Learning Using Quantum Computation

Advances in Multi-agent Reinforcement Learning: Persistent Autonomy and Robot Learning Lab Report 2024

Multi-Agent Quantum Reinforcement Learning using Evolutionary Optimization

Safe Multi-Agent Reinforcement Learning with Bilevel Optimization in Autonomous Driving

Robust Multi-Agent Control via Maximum Entropy Heterogeneous-Agent Reinforcement Learning

eQMARL: Entangled Quantum Multi-Agent Reinforcement Learning for Distributed Cooperation over Quantum Channels

MQE: Unleashing the Power of Interaction with Multi-agent Quadruped Environment

Safe reinforcement learning under temporal logic with reward design and quantum action selection

Short quantum circuits in reinforcement learning policies for the vehicle routing problem