Abstract:In real-world multi-agent reinforcement learning (MARL) applications, agents may not have perfect state information (e.g., due to inaccurate measurement or malicious attacks), which challenges the robustness of agents' policies. Though robustness is getting important in MARL deployment, little prior work has studied state uncertainties in MARL, neither in problem formulation nor algorithm design. Motivated by this robustness issue and the lack of corresponding studies, we study the problem of MARL with state uncertainty in this work. We provide the first attempt to the theoretical and empirical analysis of this challenging problem. We first model the problem as a Markov Game with state perturbation adversaries (MG-SPA) by introducing a set of state perturbation adversaries into a Markov Game. We then introduce robust equilibrium (RE) as the solution concept of an MG-SPA. We conduct a fundamental analysis regarding MG-SPA such as giving conditions under which such a robust equilibrium exists. Then we propose a robust multi-agent Q-learning (RMAQ) algorithm to find such an equilibrium, with convergence guarantees. To handle high-dimensional state-action space, we design a robust multi-agent actor-critic (RMAAC) algorithm based on an analytical expression of the policy gradient derived in the paper. Our experiments show that the proposed RMAQ algorithm converges to the optimal value function; our RMAAC algorithm outperforms several MARL and robust MARL methods in multiple multi-agent environments when state uncertainty is present. The source code is public on \url{<a class="link-external link-https" href="https://github.com/sihongho/robust_marl_with_state_uncertainty" rel="external noopener nofollow">this https URL</a>}.

What problem does this paper attempt to address?

This paper attempts to solve the policy robustness problem in Multi - Agent Reinforcement Learning (MARL) caused by state uncertainty. In practical applications, agents may not be able to obtain complete state information, which may be due to inaccurate measurement or malicious attacks, etc. This state uncertainty poses a challenge to the robustness of agent policies. Although robustness is becoming more and more important in the practical deployment of MARL, previous studies have rarely focused on the state uncertainty problem in MARL, whether from the perspective of problem modeling or algorithm design. To address this robustness problem and fill the relevant research gap, the authors studied the MARL problem with state uncertainty in this work. They made the first attempt to conduct theoretical and empirical analysis of this problem. Specifically, the authors first modeled the problem as a Markov Game with State Perturbation Adversaries (MG - SPA) by introducing a set of state perturbation adversaries. Then, they introduced "Robust Equilibrium" (RE) as the solution concept of MG - SPA. In addition, the authors also carried out basic analysis on MG - SPA, such as giving the conditions for the existence of such robust equilibria. Next, a Robust Multi - Agent Q - learning (RMAQ) algorithm was proposed to find such equilibria and convergence guarantees were provided. To handle high - dimensional state - action spaces, based on the analytical expression of the policy gradient derived in the paper, a Robust Multi - Agent Actor - Critic (RMAAC) algorithm was designed. The experimental results show that the proposed RMAQ algorithm can converge to the optimal value function; the RMAAC algorithm outperforms several MARL and robust MARL methods in multiple multi - agent environments in the presence of state uncertainty. These contributions mark important progress in the MARL problem with state uncertainty both theoretically and practically.

Robust Multi-Agent Reinforcement Learning with State Uncertainty

What is the Solution for State-Adversarial Multi-Agent Reinforcement Learning?

S2rl

S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

Robust Multi-Agent Control via Maximum Entropy Heterogeneous-Agent Reinforcement Learning

Robust Multi-Agent Reinforcement Learning by Mutual Information Regularization

MIR2: Towards Provably Robust Multi-Agent Reinforcement Learning by Mutual Information Regularization

Robustness Testing for Multi-Agent Reinforcement Learning: State Perturbations on Critical Agents

Breaking the Curse of Multiagency in Robust Multi-Agent Reinforcement Learning

Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms

Robust Multiobjective Reinforcement Learning Considering Environmental Uncertainties

Robust Multi-Agent Reinforcement Learning Driven by Correlated Equilibrium

Robustness to Multi-Modal Environment Uncertainty in MARL using Curriculum Learning

Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game

On Practical Robust Reinforcement Learning: Adjacent Uncertainty Set and Double-Agent Algorithm.

Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertainty

Safe Multi-Agent Reinforcement Learning for Multi-Robot Control

Robust Cooperative Multi-Agent Reinforcement Learning:A Mean-Field Type Game Perspective

Safe Multi-Agent Reinforcement Learning with Convergence to Generalized Nash Equilibrium

RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization

Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations