Abstract:Effective exploration is crucial to discovering optimal strategies for multi-agent reinforcement learning (MARL) in complex coordination tasks. Existing methods mainly utilize intrinsic rewards to enable committed exploration or use role-based learning for decomposing joint action spaces instead of directly conducting a collective search in the entire action-observation space. However, they often face challenges obtaining specific joint action sequences to reach successful states in long-horizon tasks. To address this limitation, we propose Imagine, Initialize, and Explore (IIE), a novel method that offers a promising solution for efficient multi-agent exploration in complex scenarios. IIE employs a transformer model to imagine how the agents reach a critical state that can influence each other's transition functions. Then, we initialize the environment at this state using a simulator before the exploration phase. We formulate the imagination as a sequence modeling problem, where the states, observations, prompts, actions, and rewards are predicted autoregressively. The prompt consists of timestep-to-go, return-to-go, influence value, and one-shot demonstration, specifying the desired state and trajectory as well as guiding the action generation. By initializing agents at the critical states, IIE significantly increases the likelihood of discovering potentially important under-explored regions. Despite its simplicity, empirical results demonstrate that our method outperforms multi-agent exploration baselines on the StarCraft Multi-Agent Challenge (SMAC) and SMACv2 environments. Particularly, IIE shows improved performance in the sparse-reward SMAC tasks and produces more effective curricula over the initialized states than other generative methods, such as CVAE-GAN and diffusion models.

CIExplore: Curiosity and Influence-based Exploration in Multi-Agent Cooperative Scenarios with Sparse Rewards

CMBE: Curiosity-driven Model-Based Exploration for Multi-Agent Reinforcement Learning in Sparse Reward Settings

Two Heads Are Better Than One: A Simple Exploration Framework for Efficient Multi-Agent Reinforcement Learning.

Curiosity-driven Exploration in Sparse-reward Multi-agent Reinforcement Learning

Episodic Multi-agent Reinforcement Learning with Curiosity-driven Exploration

Individual Contributions as Intrinsic Exploration Scaffolds for Multi-agent Reinforcement Learning

Optimal Exploration Algorithm of Multi-Agent Reinforcement Learning Methods (Student Abstract)

Neighborhood Curiosity-Based Exploration in Multi-Agent Reinforcement Learning

Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning

Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning

Influence-Based Multi-Agent Exploration

Improving Cooperative Multi-Agent Exploration via Surprise Minimization and Social Influence Maximization

Situation-Dependent Causal Influence-Based Cooperative Multi-agent Reinforcement Learning

Subspace-Aware Exploration for Sparse-Reward Multi-Agent Tasks.

Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain

Attention-based Curiosity-driven Exploration in Deep Reinforcement Learning

Curiosity Driven Collaborative Reconnaissance of Multiple Unmanned Aerial Vehicles

A Fuzzy Curiosity-Driven Mechanism for Multi-Agent Reinforcement Learning.

CPIG: Leveraging Consistency Policy with Intention Guidance for Multi-agent Exploration

Random curiosity-driven exploration in deep reinforcement learning

Self-Motivated Multi-Agent Exploration