Abstract:Uncertainty in the evolution of opponent behavior creates a non-stationary environment for the agent, reducing the reliability of value estimation and strategy selection while compromising security during the exploration process. Previous studies have developed various uncertainty quantification techniques and designed uncertainty-aware exploration methods for multi-agent reinforcement learning (MARL). However, existing methods have gaps in theoretical research and experimental verification of decoupling uncertainty between opponents and environment, which can decrease learning efficiency and lead to an unstable training process. Due to inaccurate opponent modeling, the agent is vulnerable to harm from opponents, which is undesirable in real-world tasks. To address these issues, this study proposes a novel uncertainty-guided safe exploration strategy for MARL that decouples the two types of uncertainty originating from the environment and opponents. Specifically, we introduce an uncertainty decoupling quantification technique based on a novel variance decomposition method for action-value functions. Furthermore, we present an uncertainty-aware policy optimization mechanism to facilitate safe exploration in MARL. Finally, we propose a new adaptive parameter scaling method to ensure efficient exploration by the agents. Theoretical analysis establishes the proposed approach’s convergence rate, and its effectiveness is demonstrated empirically. Extensive experiments on benchmark tasks spanning differential games, multi-agent particle environments, and RoboSumo validate the proposed uncertainty-guided method’s significant advantages in attaining higher scores and facilitating safe agent exploration.

An Uncertainty Decision Model for Partial Observable Agents by Dynamically Sharing Group Utilities

Optimal Travel Information Provision Strategies: an Agent-Based Approach under Uncertainty

A Game-Theoretic Utility Network for Cooperative Multi-Agent Decisions in Adversarial Environments

Uncertain agency models with multi-dimensional incomplete information based on confidence level

A Hierarchical Game-Theoretic Decision-Making for Cooperative Multi-Agent Systems Under the Presence of Adversarial Agents

Modeling Agents Combined With Environment, Mental States And Utility

Artificial Decision Making Under Uncertainty in Intelligent Buildings

Diversifying Agent's Behaviors in Interactive Decision Models

Integrating Decision Sharing with Prediction in Decentralized Planning for Multi-Agent Coordination under Uncertainty.

A Multi-Agent Game for Studying Human Decision-Making

A Multi-Agent Model for the Reasoning of Uncertainty Information in Supply Chains

Multiagent Multiobjective Decision Making and Game for Saving Public Resources

A Unified Approach to Dynamic Decision Problems with Asymmetric Information - Part I: Non-Strategic Agents

Modeling Human-Like Decision Making for Virtual Agents in Time-Critical Situations

Decentralized Opportunistic Spectrum Resources Access Model and Algorithm Toward Cooperative Ad-Hoc Networks

Multi-Agent Automated Negotiation Model under Uncertainty

Bridging the Gap between Partially Observable Stochastic Games and Sparse POMDP Methods

A Role-Based POMDPs Approach for Decentralized Implicit Cooperation of Multiple Agents.

Utility-Maximizing Task Scheduling for Partially Observable Multiagent Systems

A multi-agent model of hierarchical decision dynamics

Uncertainty Modified Policy for Multi-Agent Reinforcement Learning