Abstract: Centralized Training with Decentralized Execution (CTDE) has recently emerged as a popular framework for cooperative Multi-Agent Reinforcement Learning (MARL), where agents can use additional global state information to guide training in a centralized way and make their own decisions only based on decentralized local policies. Despite the encouraging results achieved, CTDE makes an independence assumption on agent policies, which limits agents to adopt global cooperative information from each other during centralized training. Therefore, we argue that existing CTDE methods cannot fully utilize global information for training, leading to an inefficient joint-policy exploration and even suboptimal results. In this paper, we introduce a novel Centralized Advising and Decentralized Pruning (CADP) framework for multi-agent reinforcement learning, that not only enables an efficacious message exchange among agents during training but also guarantees the independent policies for execution. Firstly, CADP endows agents the explicit communication channel to seek and take advices from different agents for more centralized training. To further ensure the decentralized execution, we propose a smooth model pruning mechanism to progressively constraint the agent communication into a closed one without degradation in agent cooperation capability. Empirical evaluations on StarCraft II micromanagement and Google Research Football benchmarks demonstrate that the proposed framework achieves superior performance compared with the state-of-the-art counterparts. Our code will be made publicly available.

QDAP: Downsizing adaptive policy for cooperative multi-agent reinforcement learning

A Dynamically Adaptive Approach to Reducing Strategic Interference for Multi-agent Systems

Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?

Dueling Network Architecture for Multi-Agent Deep Deterministic Policy Gradient

Q-value Path Decomposition for Deep Multiagent Reinforcement Learning

Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning

ACE: Cooperative Multi-agent Q-learning with Bidirectional Action-Dependency

Consciousness-Aware Multi-Agent Reinforcement Learning

Learning Multi-Agent Cooperation via Considering Actions of Teammates

Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learning

Decentralized Reinforcement Social Learning Based on Cooperative Policy Exploration in Multi-Agent Systems.

Multi-Agent Determinantal Q-Learning

Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces

Multi-agent Dueling Q-learning with Mean Field and Value Decomposition

Qauxi: Cooperative Multi-Agent Reinforcement Learning with Knowledge Transferred from Auxiliary Task

Offline Decentralized Multi-Agent Reinforcement Learning

DDMA: Discrepancy-Driven Multi-agent Reinforcement Learning

Reducing overestimation in value mixing for cooperative deep multi-agent reinforcement learning

A Distributed Q-Learning Algorithm for Multi-Agent Team Coordination

Multiagent Q-learning with Sub-Team Coordination.

QPLEX: Duplex Dueling Multi-Agent Q-Learning.