Abstract:In real-world multi-agent reinforcement learning (MARL) applications, agents may not have perfect state information (e.g., due to inaccurate measurement or malicious attacks), which challenges the robustness of agents' policies. Though robustness is getting important in MARL deployment, little prior work has studied state uncertainties in MARL, neither in problem formulation nor algorithm design. Motivated by this robustness issue and the lack of corresponding studies, we study the problem of MARL with state uncertainty in this work. We provide the first attempt to the theoretical and empirical analysis of this challenging problem. We first model the problem as a Markov Game with state perturbation adversaries (MG-SPA) by introducing a set of state perturbation adversaries into a Markov Game. We then introduce robust equilibrium (RE) as the solution concept of an MG-SPA. We conduct a fundamental analysis regarding MG-SPA such as giving conditions under which such a robust equilibrium exists. Then we propose a robust multi-agent Q-learning (RMAQ) algorithm to find such an equilibrium, with convergence guarantees. To handle high-dimensional state-action space, we design a robust multi-agent actor-critic (RMAAC) algorithm based on an analytical expression of the policy gradient derived in the paper. Our experiments show that the proposed RMAQ algorithm converges to the optimal value function; our RMAAC algorithm outperforms several MARL and robust MARL methods in multiple multi-agent environments when state uncertainty is present. The source code is public on \url{<a class="link-external link-https" href="https://github.com/sihongho/robust_marl_with_state_uncertainty" rel="external noopener nofollow">this https URL</a>}.

Robust Active Measuring under Model Uncertainty

Robust Markov Decision Processes without Model Estimation

Multistage Robust Mixed-Integer Optimization under Endogenous Uncertainty

Robust Anytime Learning of Markov Decision Processes

Robust Markov Decision Processes: A Place Where AI and Formal Methods Meet

Avoiding Model Estimation in Robust Markov Decision Processes with a Generative Model

Sequential Decision-Making under Uncertainty: A Robust MDPs review

Tractable Robust Markov Decision Processes

Robustness to Modeling Errors in Risk-Sensitive Markov Decision Problems with Markov Risk Measures

Model-Free Robust Average-Reward Reinforcement Learning

Robust Model-Based Reinforcement Learning with an Adversarial Auxiliary Model

Sample Complexity of Robust Reinforcement Learning with a Generative Model

Learning Robust Policies for Uncertain Parametric Markov Decision Processes

Decision-Making Under Uncertainty: Beyond Probabilities

Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes

Time-Constrained Robust MDPs

Robust Multi-Agent Reinforcement Learning with State Uncertainty

Robust Average-Reward Markov Decision Processes

Markov Decision Processes with Noisy State Observation

Roping in Uncertainty: Robustness and Regularization in Markov Games

Policy Learning for Robust Markov Decision Process with a Mismatched Generative Model